Oct 30 21:36:44.817023 ip-10-0-133-193 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:36:44.817039 ip-10-0-133-193 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:36:44.817048 ip-10-0-133-193 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:36:44.817375 ip-10-0-133-193 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:36:54.923909 ip-10-0-133-193 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:36:54.923925 ip-10-0-133-193 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9f2d6868facc4992bd22a29288388f8d -- Oct 30 21:38:52.701913 ip-10-0-133-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:38:53.108517 ip-10-0-133-193 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:53.108517 ip-10-0-133-193 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:38:53.108517 ip-10-0-133-193 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:53.108517 ip-10-0-133-193 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:53.113784 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113623 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.113784 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113784 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113790 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113795 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113798 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113802 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113805 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113808 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113810 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113813 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113816 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113818 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113821 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113824 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113826 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113829 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113831 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113834 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113837 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113839 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.113863 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113845 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113847 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113850 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113853 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113855 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113858 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113869 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113873 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113876 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113879 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113882 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113884 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113887 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113889 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113892 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113894 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113897 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113900 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113902 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.114296 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113905 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113907 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113910 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113913 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113915 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113918 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113921 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113923 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113926 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113929 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113931 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113934 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113936 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113939 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113942 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113945 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113947 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113950 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113953 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.114772 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113956 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113959 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113961 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113965 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113967 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113970 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113972 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113975 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113977 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113980 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113982 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113985 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113988 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113991 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113993 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113997 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.113999 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114003 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114006 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114010 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.115220 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114013 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114016 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114018 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114021 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114025 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114027 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114030 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114033 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114036 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114039 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114041 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114046 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114050 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114053 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114056 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114059 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114061 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114064 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114067 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114069 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.115718 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114072 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114074 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114077 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114079 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114082 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114086 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114089 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114895 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114903 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114906 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114910 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114913 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114916 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114919 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114922 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114925 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114927 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114930 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114933 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114936 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.116218 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114939 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114942 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114944 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114947 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114950 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114953 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114956 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114958 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114961 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114964 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114967 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114970 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114973 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114975 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114978 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114981 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114983 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114986 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114988 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114991 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.116703 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114993 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114996 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.114999 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115001 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115004 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115007 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115010 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115012 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115015 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115018 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115020 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115023 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115026 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115029 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115031 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115034 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115037 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115039 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115042 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115044 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.117229 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115047 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115049 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115052 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115055 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115058 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115062 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115065 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115068 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115072 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115074 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115077 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115080 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115082 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115085 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115087 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115090 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115092 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115096 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115101 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.117735 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115104 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115107 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115110 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115114 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115118 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115120 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115123 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115125 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115128 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115131 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115133 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115136 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115139 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115141 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115144 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115147 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115149 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115152 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115155 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.118182 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115157 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115160 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115162 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115165 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115167 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115185 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115189 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115192 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115196 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115199 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115206 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115209 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115212 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115215 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115728 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115734 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115739 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115742 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.118698 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115745 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115747 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115750 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115753 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115756 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115759 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115761 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115764 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115766 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115769 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115771 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115774 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115776 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115779 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115781 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115784 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115787 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115790 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115793 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115795 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.119155 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115798 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115800 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115803 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115806 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115809 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115811 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115817 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115820 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115822 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115825 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115828 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115831 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115834 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115836 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115839 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115842 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115845 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115847 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115850 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.119649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115852 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115855 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115858 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115860 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115863 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115865 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115868 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115870 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115874 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115876 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115879 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115881 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115884 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115887 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115889 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115892 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115895 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115897 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115900 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.120122 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115903 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115907 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115909 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115912 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115915 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115919 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115921 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115924 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115927 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115929 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115932 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115934 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115937 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115939 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115942 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115944 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115950 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115952 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115955 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.120569 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115957 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115960 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115962 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115965 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115967 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115971 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115973 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115977 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115981 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115983 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115986 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115989 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115992 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115995 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.115998 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116003 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116007 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116010 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116013 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.121054 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116015 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116018 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116020 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.116023 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117127 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117137 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117144 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117149 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117153 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117157 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117161 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117166 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117169 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117172 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117176 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117179 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117182 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117185 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117188 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117192 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117195 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117198 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117203 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:38:53.121484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117206 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117210 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117213 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117217 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117222 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117225 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117229 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117232 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117235 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117239 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117242 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117245 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117248 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117253 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117256 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117259 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117262 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117265 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117268 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117273 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117276 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117279 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117282 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117285 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:38:53.122061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117290 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117293 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117296 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117299 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117303 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117306 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117309 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117312 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117316 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117319 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117324 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117329 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117332 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117336 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117340 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117343 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117346 2576 flags.go:64] FLAG: --help="false" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117349 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117352 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117355 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117358 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117362 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117365 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:38:53.122706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117368 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117371 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117375 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117378 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117381 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117384 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117387 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117390 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117393 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117397 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117400 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117403 2576 flags.go:64] FLAG: --lock-file="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117406 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117409 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117412 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117418 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117421 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117424 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117427 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117431 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117434 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117437 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117440 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117446 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117449 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:38:53.123260 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117453 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117456 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117459 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117462 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117466 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117469 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117472 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117475 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117482 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117485 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117488 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117491 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117495 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117498 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117501 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117504 2576 flags.go:64] FLAG: --port="10250" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117507 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117510 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c7b723c4fdace55b" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117513 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117518 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117522 2576 flags.go:64] FLAG: --register-node="true" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117525 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117529 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117532 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117535 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117538 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:38:53.123957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117543 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117546 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117549 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117553 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117556 2576 flags.go:64] FLAG: --runonce="false" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117560 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117563 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117566 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117569 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117572 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117576 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117579 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117582 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117585 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117588 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117591 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117594 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117597 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117600 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117603 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117608 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117611 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117614 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117619 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117622 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:38:53.124598 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117626 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117646 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117651 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117656 2576 flags.go:64] FLAG: --v="2" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117661 2576 flags.go:64] FLAG: --version="false" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117666 2576 flags.go:64] FLAG: --vmodule="" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117670 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.117675 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117812 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117816 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117820 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117824 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117828 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117832 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117834 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117837 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117840 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117842 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117845 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117847 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117850 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.125199 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117853 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117855 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117858 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117861 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117864 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117866 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117869 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117871 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117874 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117876 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117879 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117883 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117885 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117888 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117890 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117893 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117895 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117898 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117902 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.125701 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117904 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117907 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117910 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117912 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117915 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117917 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117920 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117923 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117926 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117928 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117931 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117934 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117936 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117939 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117941 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117944 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117949 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117952 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117955 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.126159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117957 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117962 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117965 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117969 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117972 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117975 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117977 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117980 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117983 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117985 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117989 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117992 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117995 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.117998 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118002 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118005 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118008 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118011 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.126650 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118014 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118017 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118020 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118022 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118025 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118028 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118030 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118033 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118036 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118038 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118041 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118044 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118046 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118048 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118051 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118054 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118056 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118060 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118062 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118065 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.127111 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118068 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118070 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118073 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118075 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118079 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118082 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118085 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118087 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118090 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118092 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118095 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118098 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118101 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118104 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118107 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.118132 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:53.127612 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118169 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118174 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118177 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118180 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118183 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118186 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118189 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118193 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118195 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118199 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118203 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118207 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118210 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118214 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118217 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118219 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118222 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118225 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118227 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.128012 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118230 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118232 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118236 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118238 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118241 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118243 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118246 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118248 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118251 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118254 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118256 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118259 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118262 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118265 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118267 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118270 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118273 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118275 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118279 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118281 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.128511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118284 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118286 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118289 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118291 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118294 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118296 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118300 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118302 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118305 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118307 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118310 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118313 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118317 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118319 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118322 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118325 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118327 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118330 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118333 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118335 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.129310 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118338 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118340 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118343 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118346 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118349 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118352 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118354 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118357 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118359 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118362 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118364 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118367 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118369 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118372 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118374 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118377 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118379 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118382 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118385 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.129934 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118388 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118391 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118393 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118396 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118398 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118401 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118403 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118406 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118408 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118411 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118413 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118416 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118418 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118421 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118423 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118426 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118428 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118431 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118434 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118437 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.130728 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118439 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118442 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118444 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118447 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118449 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118452 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118454 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.118458 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.125975 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.125992 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.126009 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.126014 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126071 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126080 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126084 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126088 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126091 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126094 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.131340 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126097 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126100 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126102 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126105 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126108 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126111 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126113 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126116 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126119 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126121 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126124 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126127 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126129 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126132 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126137 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126140 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126143 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126145 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126148 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.131916 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126150 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126153 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126157 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126160 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126163 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126166 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126168 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126171 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126174 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126176 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126179 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126181 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126186 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126189 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126191 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126194 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126196 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126199 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126202 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.132431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126204 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126207 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126209 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126212 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126214 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126217 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126220 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126222 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126227 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126231 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126233 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126236 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126240 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126242 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126245 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126248 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126251 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126254 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126256 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.132937 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126259 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126261 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126264 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126266 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126269 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126271 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126274 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126277 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126279 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126282 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126284 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126287 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126290 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126292 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126295 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126298 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126300 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126303 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126305 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.133431 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126308 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126311 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126313 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126316 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126319 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126321 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126324 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126326 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126330 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126333 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126336 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126339 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126341 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126343 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126346 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126349 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126351 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126354 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126356 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.134016 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126359 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126361 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.126389 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126544 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126549 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126552 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126555 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126558 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126560 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126563 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126566 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126569 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126571 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126574 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126577 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126580 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.134455 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126582 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126585 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126587 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126590 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126592 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126595 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126597 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126600 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126603 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126606 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126608 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126611 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126613 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126616 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126618 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126621 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126623 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126626 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126642 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.135019 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126645 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126648 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126650 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126653 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126656 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126659 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126662 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126666 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126669 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126672 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126675 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126679 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126681 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126684 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126687 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126689 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126692 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126695 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126697 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.135807 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126700 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126702 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126705 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126709 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126713 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126716 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126718 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126721 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126724 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126726 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126729 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126731 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126734 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126737 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126739 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126742 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126744 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126746 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126749 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.136589 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126751 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126754 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126756 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126759 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126761 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126772 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126775 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126777 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126780 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126782 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126785 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126787 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126790 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126792 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126795 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126797 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126800 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126803 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126805 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.137083 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126808 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126811 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126813 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126816 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126818 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126821 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126823 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126826 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126828 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126831 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126833 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126836 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126838 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126841 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126843 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126846 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.137531 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126849 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.126871 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126902 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126907 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126909 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126912 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126915 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126917 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126920 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126923 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126925 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126928 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126930 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126933 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126935 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126938 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:53.137938 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126941 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126943 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126946 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126949 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126951 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126954 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126957 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126959 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126962 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126964 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126967 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126969 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126972 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126974 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126977 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126979 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126981 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126984 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126986 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:53.138303 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126989 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126992 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126995 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.126997 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127000 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127003 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127006 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127010 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127013 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127016 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127019 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127021 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127024 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127027 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127030 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127032 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127035 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127037 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127040 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:53.138891 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127042 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127044 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127048 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127052 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127055 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127057 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127060 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127062 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127067 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127071 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127074 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127076 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127079 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127081 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127084 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127086 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127089 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127092 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:53.139649 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127094 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127097 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127100 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127102 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127105 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127107 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127110 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127112 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127115 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127118 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127121 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127123 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127126 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127131 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127133 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127135 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127138 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127140 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:53.140112 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127143 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127145 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127148 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127152 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127156 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127159 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127162 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127164 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127167 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127169 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127172 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127174 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127177 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127180 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127183 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:53.127186 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.127255 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.131792 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.132941 2576 server.go:1014] "Starting client certificate rotation" Oct 30 21:38:53.140590 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.133041 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:38:53.141141 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.133085 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:38:53.141141 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.135672 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:38:53.141141 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.139592 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:38:53.153775 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.153755 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:38:53.156743 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.156729 2576 log.go:25] "Validated CRI v1 image API" Oct 30 21:38:53.157521 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.157494 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:38:53.160245 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.160222 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e7036303-b278-4b36-890d-499b248ed4b4:/dev/nvme0n1p4 f1e73ae8-eef3-44eb-a528-72dadcc58f68:/dev/nvme0n1p3] Oct 30 21:38:53.160320 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.160243 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:38:53.164843 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.164826 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:38:53.165691 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.165558 2576 manager.go:217] Machine: {Timestamp:2026-10-30 21:38:53.164092877 +0000 UTC m=+0.362491222 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101871 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f7fb92e2bc44abd82e05f5e33d17 SystemUUID:ec25f7fb-92e2-bc44-abd8-2e05f5e33d17 BootID:9f2d6868-facc-4992-bd22-a29288388f8d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:ee:a3:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:ee:a3:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:59:d3:cf:9b:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:38:53.165779 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.165692 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:38:53.165829 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.165800 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:38:53.169407 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.169385 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:38:53.169535 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.169409 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-193.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:38:53.170254 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.170244 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:38:53.170286 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.170256 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:38:53.170286 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.170280 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:38:53.170364 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.170292 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:38:53.171642 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.171620 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:38:53.176344 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176332 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:38:53.176455 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176445 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176499 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176508 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176546 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176556 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.176570 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:38:53.177253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.177028 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:38:53.181243 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.181219 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:38:53.187812 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.187796 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:38:53.188678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188664 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:38:53.188733 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188696 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:38:53.188873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188862 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:38:53.188904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188880 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:38:53.188904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188887 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:38:53.188904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188892 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:38:53.188904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188898 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:38:53.189007 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.188907 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:38:53.189603 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189593 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:38:53.189678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189605 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:38:53.189678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189612 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:38:53.189678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189619 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:38:53.189678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189647 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:38:53.189678 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.189662 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:38:53.190267 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.190257 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:38:53.190267 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.190267 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:38:53.194550 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.194537 2576 server.go:1263] "Started kubelet" Oct 30 21:38:53.194711 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.194651 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:38:53.194770 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.194730 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:38:53.194905 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.194876 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:38:53.195230 ip-10-0-133-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:38:53.195893 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.195758 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:38:53.195893 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.195808 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-193.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:38:53.196059 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.195872 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-193.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:38:53.196307 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.196246 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:38:53.197022 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.197006 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:38:53.201013 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.200991 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:38:53.203773 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.203754 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:38:53.204324 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.204308 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:38:53.204479 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.204434 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.205086 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205070 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:38:53.205086 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205076 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:38:53.205201 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205093 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:38:53.205357 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205344 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:38:53.205392 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205359 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:38:53.205871 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.205851 2576 factory.go:55] Registering systemd factory Oct 30 21:38:53.206387 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206368 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:38:53.206697 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206681 2576 factory.go:153] Registering CRI-O factory Oct 30 21:38:53.206697 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206698 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 21:38:53.206846 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206758 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:38:53.206846 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206786 2576 factory.go:103] Registering Raw factory Oct 30 21:38:53.206846 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.206799 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 21:38:53.207199 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.205964 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-193.us-east-2.compute.internal.18e36c76958bfc64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-193.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-193.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:38:53.194509412 +0000 UTC m=+0.392907757,LastTimestamp:2026-10-30 21:38:53.194509412 +0000 UTC m=+0.392907757,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-193.us-east-2.compute.internal,}" Oct 30 21:38:53.207827 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.207808 2576 manager.go:319] Starting recovery of all containers Oct 30 21:38:53.209936 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.209914 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-193.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:38:53.219892 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.219871 2576 manager.go:324] Recovery completed Oct 30 21:38:53.224200 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.224186 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.224484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.224464 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zkgrv" Oct 30 21:38:53.224933 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.224916 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:38:53.225957 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.225938 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.226032 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.225966 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.226032 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.225976 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.226410 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.226399 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:38:53.226410 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.226408 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:38:53.226522 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.226428 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:38:53.226522 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.226409 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.227220 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.227206 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.227277 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.227231 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.227277 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.227242 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.228701 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.228689 2576 policy_none.go:50] "Start" Oct 30 21:38:53.228755 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.228704 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:38:53.228755 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.228714 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:38:53.229259 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.229172 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-193.us-east-2.compute.internal.18e36c76976bc8e8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-193.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-193.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-193.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:38:53.225953512 +0000 UTC m=+0.424351857,LastTimestamp:2026-10-30 21:38:53.225953512 +0000 UTC m=+0.424351857,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-193.us-east-2.compute.internal,}" Oct 30 21:38:53.230735 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.230723 2576 policy_none.go:44] "Start" Oct 30 21:38:53.235463 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.235355 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zkgrv" Oct 30 21:38:53.239288 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.239229 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-193.us-east-2.compute.internal.18e36c76976c0b0a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-193.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-193.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-193.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:38:53.225970442 +0000 UTC m=+0.424368787,LastTimestamp:2026-10-30 21:38:53.225970442 +0000 UTC m=+0.424368787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-193.us-east-2.compute.internal,}" Oct 30 21:38:53.271259 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.271235 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:38:53.273479 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.273467 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 21:38:53.273531 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.273501 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:38:53.273531 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.273510 2576 server.go:87] "Starting device plugin registration server" Oct 30 21:38:53.273772 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.273760 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:38:53.273804 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.273774 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:38:53.273963 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.273912 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:38:53.274063 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.274041 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:38:53.274063 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.274052 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:38:53.274716 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.274693 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:38:53.274797 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.274745 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.374654 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.374571 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.376406 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.375600 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.376406 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.375641 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.376406 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.375666 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.376406 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.375707 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.384676 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.384659 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.384759 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.384681 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-193.us-east-2.compute.internal\": node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.386709 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.386688 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:38:53.388034 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.388012 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:38:53.388106 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.388061 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:38:53.388106 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.388089 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:38:53.388182 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.388126 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:38:53.392940 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.392924 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:53.488347 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.488312 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal"] Oct 30 21:38:53.488443 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.488402 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.489271 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.489257 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.489326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.489288 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.489326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.489298 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.491393 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.491381 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.491516 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.491502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.491562 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.491533 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.492106 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492082 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.492207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492116 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.492207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492087 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.492207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492139 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.492207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492147 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.492207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.492159 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.494185 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.494169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.494265 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.494198 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:53.495021 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.495008 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:53.495099 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.495036 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:53.495099 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.495058 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:53.507471 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.507449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.507559 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.507473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.507559 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.507489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.521822 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.521802 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.524809 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.524793 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.578570 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.578556 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.608234 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.608304 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.608304 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.608365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.608365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.608365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.608357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.678667 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.678606 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.779176 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.779150 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.825603 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.825581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.828750 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:53.828734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:53.879696 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.879666 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:53.980315 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:53.980236 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.080706 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.080681 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.132130 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.132109 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:38:54.132506 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.132261 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:38:54.181644 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.181605 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.239993 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.239918 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:54.240936 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.240892 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:33:54 +0000 UTC" deadline="2028-03-27 12:25:08.474712389 +0000 UTC" Oct 30 21:38:54.240936 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.240933 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:54.241062 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.240964 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:54.241062 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.240975 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:54.241062 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.240935 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12326h46m14.23378024s" Oct 30 21:38:54.282709 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.282685 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.323861 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.323831 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:54.383092 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.383064 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.411924 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.411902 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:54.458516 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:54.458489 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05953060daddf043bb6b311d0281468b.slice/crio-a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5 WatchSource:0}: Error finding container a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5: Status 404 returned error can't find the container with id a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5 Oct 30 21:38:54.459159 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:54.459142 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9317762339865a73d28250c9a6a406c.slice/crio-fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803 WatchSource:0}: Error finding container fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803: Status 404 returned error can't find the container with id fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803 Oct 30 21:38:54.463887 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.463870 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:38:54.483691 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.483666 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.584344 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.584283 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.684977 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:54.684954 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.us-east-2.compute.internal\" not found" Oct 30 21:38:54.756746 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.756727 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:54.804761 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.804742 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:54.819558 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.819541 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:54.820469 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.820457 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:38:54.828752 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:54.828729 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:56.503877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.503797 2576 apiserver.go:52] "Watching apiserver" Oct 30 21:38:56.513246 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.513220 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:38:56.513647 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.513610 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5","openshift-cluster-node-tuning-operator/tuned-slpv5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-7pq9d","openshift-image-registry/node-ca-pgsq6","openshift-dns/node-resolver-h7xwb","openshift-network-diagnostics/network-check-target-dr5t5","kube-system/global-pull-secret-syncer-nd669","kube-system/konnectivity-agent-xfnx9","openshift-multus/multus-hk8sl","openshift-multus/network-metrics-daemon-xcg97","openshift-network-operator/iptables-alerter-9qwxg","openshift-ovn-kubernetes/ovnkube-node-zfdzh"] Oct 30 21:38:56.516183 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.516157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.518422 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.518403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.520575 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520547 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.520575 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520556 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:38:56.520727 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520612 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:38:56.520727 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520649 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\"" Oct 30 21:38:56.520727 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520664 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.520873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.520830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.522007 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.521986 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.522097 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.522057 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.522148 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.522100 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:38:56.522148 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.522121 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\"" Oct 30 21:38:56.522950 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.522931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.524663 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.524553 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.524663 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.524631 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.524663 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.524637 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\"" Oct 30 21:38:56.525303 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.525285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.526416 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.526399 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.526681 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.526662 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\"" Oct 30 21:38:56.526777 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.526726 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.526777 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.526750 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:38:56.527528 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.527509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.528815 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.528794 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:38:56.528906 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.528804 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:38:56.528906 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.528829 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\"" Oct 30 21:38:56.529984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.529965 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:38:56.530931 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.530911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.531197 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.531179 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:38:56.531514 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.531500 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.531785 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.531772 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\"" Oct 30 21:38:56.532045 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.532022 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.533861 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.533842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.533958 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.533929 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:38:56.534473 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.534453 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.535463 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.535440 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.535582 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.535565 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\"" Oct 30 21:38:56.536573 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.536352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.538626 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.538514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.538626 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.538590 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:38:56.541649 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.541626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:56.541746 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.541714 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:38:56.542701 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.542793 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qsgw\" (UniqueName: \"kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.542793 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.542793 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.542793 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.542994 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.542992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsn7\" (UniqueName: \"kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cbg\" (UniqueName: \"kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.543326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhb66\" (UniqueName: \"kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.543999 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjgn\" (UniqueName: \"kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.543988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ckg\" (UniqueName: \"kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8stx\" (UniqueName: \"kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9w7\" (UniqueName: \"kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.544868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.544255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.545596 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.545211 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:38:56.569052 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.569032 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f4kbg" Oct 30 21:38:56.582691 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.582673 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f4kbg" Oct 30 21:38:56.621750 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.621723 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:38:56.621922 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.621901 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:38:56.622076 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.622059 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:38:56.622166 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.622139 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:38:56.622221 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.622168 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:38:56.622769 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.622725 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:38:56.623010 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.622994 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:38:56.623076 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.623068 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\"" Oct 30 21:38:56.623133 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.623116 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:38:56.623219 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.623192 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\"" Oct 30 21:38:56.632116 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.632094 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.644994 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsn7\" (UniqueName: \"kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.648138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cbg\" (UniqueName: \"kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645505 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhb66\" (UniqueName: \"kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.648778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.645977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646063 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649225 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646185 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646519 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjgn\" (UniqueName: \"kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.649677 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ckg\" (UniqueName: \"kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8stx\" (UniqueName: \"kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.646999 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647216 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647293 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647381 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.650205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647794 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647805 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647844 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.647911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.648428 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.648480 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:57.148462499 +0000 UTC m=+3.021226948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.650984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9w7\" (UniqueName: \"kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqt4\" (UniqueName: \"kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.648997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsgw\" (UniqueName: \"kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649611 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.651764 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649720 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.653924 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654056 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654095 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654096 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654475 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.649180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654571 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.656231 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.654798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.661848 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.661821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhb66\" (UniqueName: \"kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.662030 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.662006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjgn\" (UniqueName: \"kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.663623 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.663507 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cbg\" (UniqueName: \"kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:56.666484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.666438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8stx\" (UniqueName: \"kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.668468 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.668445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ckg\" (UniqueName: \"kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.669092 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.669050 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9w7\" (UniqueName: \"kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.669542 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.669519 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsgw\" (UniqueName: \"kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.669879 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.669857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsn7\" (UniqueName: \"kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.721983 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.721923 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerStarted","Data":"fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803"} Oct 30 21:38:56.723933 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.723908 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"05953060daddf043bb6b311d0281468b","Type":"ContainerStarted","Data":"a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5"} Oct 30 21:38:56.750173 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750296 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:56.750296 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750296 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750296 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqt4\" (UniqueName: \"kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.750512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.750543 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.750600 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:57.250582089 +0000 UTC m=+3.123346534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750803 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751006 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751010 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751027 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.750258 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.751728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751692 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.752390 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.751791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.754564 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.754503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.754564 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.754540 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.760435 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.760414 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:38:56.760525 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.760439 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:38:56.760525 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.760453 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:56.760525 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:56.760514 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:57.260495602 +0000 UTC m=+3.133260057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:56.763406 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.763388 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqt4\" (UniqueName: \"kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.830199 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.830171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk8sl" Oct 30 21:38:56.840109 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.840079 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0777e96_1263_40e3_ad73_55e2a8eaa539.slice/crio-50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95 WatchSource:0}: Error finding container 50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95: Status 404 returned error can't find the container with id 50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95 Oct 30 21:38:56.846386 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.846364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:38:56.851694 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.851670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:38:56.856679 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.856659 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7bf2d6_e040_4695_aeb3_f0bb15293c98.slice/crio-2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed WatchSource:0}: Error finding container 2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed: Status 404 returned error can't find the container with id 2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed Oct 30 21:38:56.858388 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.858340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:38:56.861878 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.861847 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74027051_12b8_4ebf_8e06_a9783a3c232c.slice/crio-e40ac18ba4a5b5de7371815b6ce2a08b09c1bde24011a10daa0f474ef77e1e02 WatchSource:0}: Error finding container e40ac18ba4a5b5de7371815b6ce2a08b09c1bde24011a10daa0f474ef77e1e02: Status 404 returned error can't find the container with id e40ac18ba4a5b5de7371815b6ce2a08b09c1bde24011a10daa0f474ef77e1e02 Oct 30 21:38:56.868368 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.868342 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921d8f4a_1e5d_4c4e_a4fe_94b920bf9b67.slice/crio-8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc WatchSource:0}: Error finding container 8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc: Status 404 returned error can't find the container with id 8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc Oct 30 21:38:56.879868 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.879724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:38:56.881904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.881885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:38:56.887850 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.887825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:38:56.891760 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.891382 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251c254_07fe_47f7_9ca3_a0e00bec385c.slice/crio-75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e WatchSource:0}: Error finding container 75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e: Status 404 returned error can't find the container with id 75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e Oct 30 21:38:56.899737 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.899023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:38:56.899737 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.899078 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c533f37_a794_4b53_8779_86fbbb312bf6.slice/crio-392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712 WatchSource:0}: Error finding container 392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712: Status 404 returned error can't find the container with id 392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712 Oct 30 21:38:56.910035 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.910011 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f4cff_6fee_4e71_b5d1_381028a7f628.slice/crio-aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407 WatchSource:0}: Error finding container aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407: Status 404 returned error can't find the container with id aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407 Oct 30 21:38:56.916639 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.916618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:38:56.927450 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:38:56.927416 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c3ccd9_9679_41a3_ba1b_c00cfd7f2a2d.slice/crio-5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef WatchSource:0}: Error finding container 5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef: Status 404 returned error can't find the container with id 5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef Oct 30 21:38:56.980310 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:56.980250 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:57.153497 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.153410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:57.153646 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.153602 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:57.153704 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.153672 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:58.153658242 +0000 UTC m=+4.026422674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:57.254623 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.254595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:57.254787 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.254768 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:57.254862 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.254848 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:58.254828025 +0000 UTC m=+4.127592475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:57.355099 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.355012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:57.355254 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.355161 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:38:57.355254 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.355181 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:38:57.355254 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.355191 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:57.355254 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.355249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:58.355236693 +0000 UTC m=+4.228001125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:57.586630 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.586333 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:33:56 +0000 UTC" deadline="2028-03-26 07:27:49.348615922 +0000 UTC" Oct 30 21:38:57.586630 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.586632 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12297h48m51.761990165s" Oct 30 21:38:57.714614 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.714545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:57.714761 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:57.714673 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:38:57.732878 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.732839 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk8sl" event={"ID":"e0777e96-1263-40e3-ad73-55e2a8eaa539","Type":"ContainerStarted","Data":"50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95"} Oct 30 21:38:57.750176 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.750140 2576 generic.go:356] "Generic (PLEG): container finished" podID="b9317762339865a73d28250c9a6a406c" containerID="7c4763f32145e84702a8bdea1def3e35b4b741cb33b8a0d0d424e3976ea41918" exitCode=0 Oct 30 21:38:57.750390 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.750228 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerDied","Data":"7c4763f32145e84702a8bdea1def3e35b4b741cb33b8a0d0d424e3976ea41918"} Oct 30 21:38:57.755687 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.755657 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h7xwb" event={"ID":"5c533f37-a794-4b53-8779-86fbbb312bf6","Type":"ContainerStarted","Data":"392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712"} Oct 30 21:38:57.767304 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.767247 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgsq6" event={"ID":"d6509794-be77-4085-aeec-92819c3c9041","Type":"ContainerStarted","Data":"d9f1e09d9f6517a4e994f7611016099cff16d476874e0f8d184520fd2bd9acc6"} Oct 30 21:38:57.773536 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.773464 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qwxg" event={"ID":"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67","Type":"ContainerStarted","Data":"8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc"} Oct 30 21:38:57.784913 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.784838 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-slpv5" event={"ID":"74027051-12b8-4ebf-8e06-a9783a3c232c","Type":"ContainerStarted","Data":"e40ac18ba4a5b5de7371815b6ce2a08b09c1bde24011a10daa0f474ef77e1e02"} Oct 30 21:38:57.791993 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.791962 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef"} Oct 30 21:38:57.806340 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.806260 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfnx9" event={"ID":"ce0f4cff-6fee-4e71-b5d1-381028a7f628","Type":"ContainerStarted","Data":"aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407"} Oct 30 21:38:57.808963 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.808893 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerStarted","Data":"75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e"} Oct 30 21:38:57.815917 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:57.815843 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" event={"ID":"da7bf2d6-e040-4695-aeb3-f0bb15293c98","Type":"ContainerStarted","Data":"2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed"} Oct 30 21:38:58.163002 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.162432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:58.163002 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.162609 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:58.163002 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.162671 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:00.162652969 +0000 UTC m=+6.035417405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:38:58.264157 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.263575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:58.264157 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.263724 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:58.264157 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.263784 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:00.263766273 +0000 UTC m=+6.136530720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:38:58.364316 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.364278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:58.364500 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.364481 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:38:58.364566 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.364502 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:38:58.364566 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.364516 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:58.364656 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.364578 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:00.36456086 +0000 UTC m=+6.237325297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:38:58.587053 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.587014 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:33:56 +0000 UTC" deadline="2028-08-16 14:34:32.272230932 +0000 UTC" Oct 30 21:38:58.587053 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.587052 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15736h55m33.685183254s" Oct 30 21:38:58.717878 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.717122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:38:58.717878 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.717251 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:38:58.717878 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.717716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:38:58.717878 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:58.717821 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:38:58.824850 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.824769 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerStarted","Data":"b589c2d51fae821a7abcf4e12fb3c983e5bbdd6b4fadb2c77cb4bee07b36a164"} Oct 30 21:38:58.831440 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.830906 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"05953060daddf043bb6b311d0281468b","Type":"ContainerStarted","Data":"9645f546bd6c79a4ef4207d53791ade632e0b93ab43ed639a4b935c3e01780e3"} Oct 30 21:38:58.863305 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.861888 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" podStartSLOduration=2.861869181 podStartE2EDuration="2.861869181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:58.861356313 +0000 UTC m=+4.734120768" watchObservedRunningTime="2026-10-30 21:38:58.861869181 +0000 UTC m=+4.734633636" Oct 30 21:38:58.863305 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:58.862134 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" podStartSLOduration=2.862123917 podStartE2EDuration="2.862123917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:58.845069378 +0000 UTC m=+4.717833832" watchObservedRunningTime="2026-10-30 21:38:58.862123917 +0000 UTC m=+4.734888372" Oct 30 21:38:59.714904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:59.714845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:38:59.715297 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:38:59.714980 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:38:59.847477 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:38:59.847440 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerStarted","Data":"cdfb3cb28a8380cfe640c097ab9ac2a608c579de40341839037ccb5b324b8de0"} Oct 30 21:39:00.191527 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.191450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:00.191762 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.191674 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:00.191762 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.191751 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.191731097 +0000 UTC m=+10.064495551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:00.292943 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.292902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:00.293103 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.293063 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:00.293180 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.293160 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.293139145 +0000 UTC m=+10.165903593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:00.393891 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.393852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:00.394067 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.394020 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:00.394067 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.394039 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:00.394067 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.394054 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:00.394239 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.394114 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.394095024 +0000 UTC m=+10.266859460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:00.717484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.716823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:00.717484 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.716961 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:00.717484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.717341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:00.717484 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:00.717431 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:00.854652 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.853831 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfnx9" event={"ID":"ce0f4cff-6fee-4e71-b5d1-381028a7f628","Type":"ContainerStarted","Data":"9ac74ef2729fb117ab555dc564dbc4d68e660bb0e988f52701d0c4a11c59a3d1"} Oct 30 21:39:00.859772 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.857473 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="cdfb3cb28a8380cfe640c097ab9ac2a608c579de40341839037ccb5b324b8de0" exitCode=0 Oct 30 21:39:00.859772 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.857521 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"cdfb3cb28a8380cfe640c097ab9ac2a608c579de40341839037ccb5b324b8de0"} Oct 30 21:39:00.872233 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:00.872145 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xfnx9" podStartSLOduration=2.221989737 podStartE2EDuration="5.872127718s" totalImagesPullingTime="3.650137981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:00.871763416 +0000 UTC m=+6.744527871" watchObservedRunningTime="2026-10-30 21:39:00.872127718 +0000 UTC m=+6.744892172" Oct 30 21:39:01.607242 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:01.607163 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:01.608068 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:01.607801 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:01.715251 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:01.714858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:01.715251 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:01.714963 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:02.716971 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.716375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:02.716971 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:02.716503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:02.716971 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.716855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:02.716971 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:02.716924 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:02.869941 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.869141 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgsq6" event={"ID":"d6509794-be77-4085-aeec-92819c3c9041","Type":"ContainerStarted","Data":"60209dae47639c78b219970d1fb79bc72fcb24a13ab5e57fdf46f171846343d4"} Oct 30 21:39:02.873249 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.872737 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:02.873249 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.873163 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" event={"ID":"da7bf2d6-e040-4695-aeb3-f0bb15293c98","Type":"ContainerStarted","Data":"d8a9871410571ec4caefbd566e6fc6752542fc718fdd23317e0d4dbaa9816ec0"} Oct 30 21:39:02.889620 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:02.889470 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pgsq6" podStartSLOduration=3.807782999 podStartE2EDuration="8.889453361s" totalImagesPullingTime="5.081670362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:02.888543181 +0000 UTC m=+8.761307636" watchObservedRunningTime="2026-10-30 21:39:02.889453361 +0000 UTC m=+8.762217817" Oct 30 21:39:03.714173 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:03.714124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:03.714374 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:03.714283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:04.230321 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.230277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:04.230797 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.230475 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:04.230797 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.230547 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:12.230527795 +0000 UTC m=+18.103292243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:04.331421 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.331384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:04.331563 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.331547 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:04.331634 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.331611 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:12.331591888 +0000 UTC m=+18.204356337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:04.358321 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.358224 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:39:04.432821 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.432751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:04.432972 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.432931 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:04.432972 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.432952 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:04.432972 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.432964 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:04.433122 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.433023 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:12.433005114 +0000 UTC m=+18.305769549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:04.608429 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.608320 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:39:04.358244414Z","UUID":"92ad55d5-5385-40d1-a59e-3391f847eb0f","Handler":null,"Name":"","Endpoint":""} Oct 30 21:39:04.609945 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.609923 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:39:04.610024 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.609951 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:39:04.715312 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.715278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:04.715444 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.715412 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:04.715842 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.715817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:04.715951 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:04.715920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:04.880604 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.880566 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" event={"ID":"da7bf2d6-e040-4695-aeb3-f0bb15293c98","Type":"ContainerStarted","Data":"c3200931144850bcf700e168fe7e09ad47abd682be8f4ae93e5a70f70557f746"} Oct 30 21:39:04.883357 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:04.883312 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk8sl" event={"ID":"e0777e96-1263-40e3-ad73-55e2a8eaa539","Type":"ContainerStarted","Data":"0e7f5de52e54d5fdf3d5a9dd66f046937445a6441fa6ee5c23f5103bfe6889b5"} Oct 30 21:39:05.714798 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.714512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:05.715196 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:05.714860 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:05.887989 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.887952 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h7xwb" event={"ID":"5c533f37-a794-4b53-8779-86fbbb312bf6","Type":"ContainerStarted","Data":"fd8800f90f7b0092a4ee991bd75af1baebc265040d8473edd655da7bcbe3de52"} Oct 30 21:39:05.890529 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.890501 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="c73bb4a005f56ea93ed118db2d888c97b383c4ec74ffaebbaf1a8be2595437c5" exitCode=0 Oct 30 21:39:05.890635 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.890589 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"c73bb4a005f56ea93ed118db2d888c97b383c4ec74ffaebbaf1a8be2595437c5"} Oct 30 21:39:05.894126 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.894103 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" event={"ID":"da7bf2d6-e040-4695-aeb3-f0bb15293c98","Type":"ContainerStarted","Data":"94b18b4017be1f6186f64cc968bb31e3cf9cf54f0508192a464ac9b2cda9d179"} Oct 30 21:39:05.905584 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.905541 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hk8sl" podStartSLOduration=4.265428119 podStartE2EDuration="11.905529077s" totalImagesPullingTime="7.640100958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:04.906550076 +0000 UTC m=+10.779314561" watchObservedRunningTime="2026-10-30 21:39:05.905529077 +0000 UTC m=+11.778293536" Oct 30 21:39:05.906445 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.906405 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h7xwb" podStartSLOduration=3.701249702 podStartE2EDuration="11.906394953s" totalImagesPullingTime="8.205145251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.905392156 +0000 UTC m=+11.778156610" watchObservedRunningTime="2026-10-30 21:39:05.906394953 +0000 UTC m=+11.779159409" Oct 30 21:39:05.927632 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:05.927592 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" podStartSLOduration=3.72676884 podStartE2EDuration="11.927580942s" totalImagesPullingTime="8.200812102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.927502875 +0000 UTC m=+11.800267329" watchObservedRunningTime="2026-10-30 21:39:05.927580942 +0000 UTC m=+11.800345396" Oct 30 21:39:06.714333 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.714307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:06.714444 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.714351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:06.714502 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:06.714449 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:06.714613 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:06.714566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:06.897608 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.897539 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qwxg" event={"ID":"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67","Type":"ContainerStarted","Data":"d8bcbac58720d5933089a0b114d5ee23a3bd29d98d3e0649473f0f2e7463192c"} Oct 30 21:39:06.900643 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.900613 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="780b7394a5c9b950b8d8f4c9422cbedf8de957693e63353049be8d9777580aa1" exitCode=0 Oct 30 21:39:06.900904 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.900877 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"780b7394a5c9b950b8d8f4c9422cbedf8de957693e63353049be8d9777580aa1"} Oct 30 21:39:06.915048 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:06.915002 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9qwxg" podStartSLOduration=5.127030446 podStartE2EDuration="12.914986694s" totalImagesPullingTime="7.787956248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:06.914835011 +0000 UTC m=+12.787599469" watchObservedRunningTime="2026-10-30 21:39:06.914986694 +0000 UTC m=+12.787751149" Oct 30 21:39:07.714832 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:07.714790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:07.714995 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:07.714927 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:07.906196 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:07.906165 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerStarted","Data":"a45eae0eeb1f0fee53e490e0c8afe6ee13c499721a635bbe1ab3164695316e25"} Oct 30 21:39:07.907940 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:07.907906 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-slpv5" event={"ID":"74027051-12b8-4ebf-8e06-a9783a3c232c","Type":"ContainerStarted","Data":"f966bf688980923296fc2c268b963d1517811547653bc73e02c1cef387ef0c5c"} Oct 30 21:39:07.954200 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:07.954157 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-slpv5" podStartSLOduration=3.935458183 podStartE2EDuration="13.954140779s" totalImagesPullingTime="10.018682596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:07.953886409 +0000 UTC m=+13.826650863" watchObservedRunningTime="2026-10-30 21:39:07.954140779 +0000 UTC m=+13.826905236" Oct 30 21:39:08.717659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:08.717631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:08.717809 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:08.717755 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:08.717866 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:08.717813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:08.717951 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:08.717929 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:08.913374 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:08.913338 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="a45eae0eeb1f0fee53e490e0c8afe6ee13c499721a635bbe1ab3164695316e25" exitCode=0 Oct 30 21:39:08.913964 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:08.913444 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"a45eae0eeb1f0fee53e490e0c8afe6ee13c499721a635bbe1ab3164695316e25"} Oct 30 21:39:09.166902 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:09.166801 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:09.167067 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:09.166956 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:09.167749 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:09.167716 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:09.714175 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:09.713941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:09.714395 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:09.714335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:10.717692 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:10.717660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:10.718080 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:10.717775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:10.718196 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:10.718178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:10.718332 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:10.718310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:11.714464 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.714304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:11.714570 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:11.714553 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:11.923378 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923346 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"0934a183b31c32b739dbc08fe76b4c6ed860fe141acd8c4cf0485609e63e2df0"} Oct 30 21:39:11.923765 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923416 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"9219abbb42d3a1c358c80c3b154e1d96f6cb9c855260683f4f1edcd3c3465422"} Oct 30 21:39:11.923765 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923454 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"836f8013ce85f9bf4cb6a2aceb4d20c2f3ff6b0bfd109d1b65197bd6984ae0a0"} Oct 30 21:39:11.923765 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923483 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"2579296460b080bd4695b2e8ba9c47ed0cf0f0d4d6b83149e7e845c89134eb3a"} Oct 30 21:39:11.923765 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923517 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"768b6ba228b40710c6fa591f363e98701be9cd21c1a3b3a4bcb72e5ac7a0e3dc"} Oct 30 21:39:11.923765 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.923547 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"f2ccd3a414fe0a3928e42dc6af57baa704015c35b1c01d983d4b6156b9765eb8"} Oct 30 21:39:11.926068 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.926046 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="a40867d735d32c11aeb909cdd89fab8b9c8babf21ff09059ef47a118b8198089" exitCode=0 Oct 30 21:39:11.926180 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:11.926090 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"a40867d735d32c11aeb909cdd89fab8b9c8babf21ff09059ef47a118b8198089"} Oct 30 21:39:12.293726 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.293645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:12.293849 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.293794 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:12.293895 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.293858 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.293841204 +0000 UTC m=+34.166605653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:12.394634 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.394606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:12.394789 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.394703 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:12.394789 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.394751 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret podName:92a37659-0e94-43e8-adb2-94ec2dfe09a1 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.39473879 +0000 UTC m=+34.267503235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret") pod "global-pull-secret-syncer-nd669" (UID: "92a37659-0e94-43e8-adb2-94ec2dfe09a1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:12.495207 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.495176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:12.495361 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.495316 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:12.495361 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.495331 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:12.495361 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.495341 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kz7s2 for pod openshift-network-diagnostics/network-check-target-dr5t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:12.495474 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.495384 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2 podName:0dc8ab91-c338-4320-b006-574a196d2333 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.495372789 +0000 UTC m=+34.368137234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz7s2" (UniqueName: "kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2") pod "network-check-target-dr5t5" (UID: "0dc8ab91-c338-4320-b006-574a196d2333") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:12.714728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.714646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:12.714849 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.714796 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:12.714894 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.714857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:12.714993 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:12.714971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:12.931873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.931843 2576 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="58adfc689c6f977598445dc2d342255582e1ba8bc8644176ba8342f5b5489c9d" exitCode=0 Oct 30 21:39:12.932321 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:12.931885 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerDied","Data":"58adfc689c6f977598445dc2d342255582e1ba8bc8644176ba8342f5b5489c9d"} Oct 30 21:39:13.714818 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:13.714778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:13.714998 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:13.714877 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:13.938061 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:13.938034 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"18a43e584bf9702bae6823d16079899aef0418ac27b5791ff36ce46691ee0987"} Oct 30 21:39:13.941984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:13.941957 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" event={"ID":"9251c254-07fe-47f7-9ca3-a0e00bec385c","Type":"ContainerStarted","Data":"48c756962a9016fec7457230dacc24e8c02b551d938240458f8a6122ceb2f99e"} Oct 30 21:39:13.971494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:13.971427 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7pq9d" podStartSLOduration=9.388521865 podStartE2EDuration="19.971414091s" totalImagesPullingTime="10.582892226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:13.969150541 +0000 UTC m=+19.841915008" watchObservedRunningTime="2026-10-30 21:39:13.971414091 +0000 UTC m=+19.844178545" Oct 30 21:39:14.715655 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:14.715618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:14.715845 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:14.715700 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:14.715845 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:14.715741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:14.715845 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:14.715791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:15.713894 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:15.713866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:15.714424 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:15.713976 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:16.714658 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.714623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:16.714658 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.714645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:16.715219 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:16.714745 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:16.715219 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:16.714834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:16.951859 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.951828 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" event={"ID":"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d","Type":"ContainerStarted","Data":"db4f6c33a66f5ecd4d4cf5c9a2a275e506b690502c168aeeca6b447767be5958"} Oct 30 21:39:16.952016 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.951979 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:16.952101 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.952088 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:16.952212 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.952192 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:16.968082 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.968023 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:16.968184 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.968128 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:16.992811 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:16.992769 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" podStartSLOduration=7.436338859 podStartE2EDuration="21.992757186s" totalImagesPullingTime="14.556418327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.991965931 +0000 UTC m=+22.864730389" watchObservedRunningTime="2026-10-30 21:39:16.992757186 +0000 UTC m=+22.865521640" Oct 30 21:39:17.694295 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.692003 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd669"] Oct 30 21:39:17.694295 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.692437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:17.694295 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:17.692586 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:17.695728 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.695700 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dr5t5"] Oct 30 21:39:17.695846 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.695813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:17.695915 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:17.695893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:17.696984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.696960 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xcg97"] Oct 30 21:39:17.697107 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:17.697061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:17.697185 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:17.697164 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:19.714088 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:19.714059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:19.714476 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:19.714059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:19.714476 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:19.714164 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:19.714476 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:19.714059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:19.714476 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:19.714227 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:19.714476 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:19.714316 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:21.714087 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:21.714055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:21.714570 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:21.714055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:21.714570 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:21.714185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:21.714570 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:21.714055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:21.714570 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:21.714223 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:21.714570 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:21.714312 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:23.714180 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:23.714146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:23.714180 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:23.714172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:23.714610 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:23.714193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:23.714610 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:23.714308 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr5t5" podUID="0dc8ab91-c338-4320-b006-574a196d2333" Oct 30 21:39:23.714610 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:23.714335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd669" podUID="92a37659-0e94-43e8-adb2-94ec2dfe09a1" Oct 30 21:39:23.714610 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:23.714415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcg97" podUID="02acb0f7-5c31-4397-ae32-fe076ef673e0" Oct 30 21:39:24.440602 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.440576 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeReady" Oct 30 21:39:24.440749 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.440681 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:39:24.482751 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.482723 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwwwz"] Oct 30 21:39:24.485842 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.485824 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7rws"] Oct 30 21:39:24.485945 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.485933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:24.488689 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.488670 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:39:24.488805 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.488789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:24.491518 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.491500 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:39:24.491625 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.491602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.496416 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.496396 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx"] Oct 30 21:39:24.496535 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.496520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.499365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.499348 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jmb6"] Oct 30 21:39:24.499484 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.499467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.502288 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.502256 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9"] Oct 30 21:39:24.502365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.502350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:24.505242 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.505223 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wh95c"] Oct 30 21:39:24.505339 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.505325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.510905 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.510884 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459"] Oct 30 21:39:24.511038 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.511006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.512753 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.512734 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:39:24.512905 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.512891 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:39:24.513029 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.513010 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:39:24.513119 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.513093 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l6nmx\"" Oct 30 21:39:24.513171 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.513133 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tqvfj\"" Oct 30 21:39:24.513823 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.513806 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.513953 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.513938 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7df45874d9-82hkt"] Oct 30 21:39:24.514055 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.514040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:24.515791 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.514386 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.515791 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.514590 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.515791 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.514607 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:39:24.517098 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.516818 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.517098 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.516885 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:39:24.518206 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.518189 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dljwb"] Oct 30 21:39:24.518348 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.518330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.520257 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.520241 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.520355 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.520241 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.520963 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.520945 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-92mlp"] Oct 30 21:39:24.521075 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.521056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.522659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.522641 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:39:24.522836 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.522821 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:39:24.524417 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.524287 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48"] Oct 30 21:39:24.524417 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.524338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.528159 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.528145 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p"] Oct 30 21:39:24.528259 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.528248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:24.529112 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.529093 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:39:24.531557 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.531539 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw"] Oct 30 21:39:24.531650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.531638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.535892 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.535874 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwwwz"] Oct 30 21:39:24.535983 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.535907 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7rws"] Oct 30 21:39:24.536046 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.535988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.538091 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.538072 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hm8lm"] Oct 30 21:39:24.545182 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.545162 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkw8"] Oct 30 21:39:24.545300 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.545285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.548536 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.548517 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wh95c"] Oct 30 21:39:24.548671 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.548610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.549149 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.549129 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:39:24.550790 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.550767 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw"] Oct 30 21:39:24.550882 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.550810 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dljwb"] Oct 30 21:39:24.551506 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.551474 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df45874d9-82hkt"] Oct 30 21:39:24.552881 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.552861 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-92mlp"] Oct 30 21:39:24.558253 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.558224 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jmb6"] Oct 30 21:39:24.563683 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.563660 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:39:24.563865 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.563842 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx"] Oct 30 21:39:24.571660 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.571638 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkw8"] Oct 30 21:39:24.573394 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.573348 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p"] Oct 30 21:39:24.575045 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.574484 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm8lm"] Oct 30 21:39:24.577185 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.577164 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48"] Oct 30 21:39:24.579796 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.579775 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459"] Oct 30 21:39:24.582497 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582605 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582605 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582605 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqh7q\" (UniqueName: \"kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q\") pod \"kas-connection-checker-685895c7c9-kwwwz\" (UID: \"9b036611-6b37-40c6-9a5d-01e3f42d6e1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582781 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9"] Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g62x\" (UniqueName: \"kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x\") pod \"kas-connection-checker-685895c7c9-r7rws\" (UID: \"98c78665-fec7-466f-a032-51b497e7028a\") " pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.582992 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.582977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.583436 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.583004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.611303 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611284 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:39:24.611508 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611489 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:39:24.611643 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611538 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:39:24.611688 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611651 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:39:24.611735 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611717 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:39:24.611735 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611726 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xzsv2\"" Oct 30 21:39:24.611823 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611734 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2mzw9\"" Oct 30 21:39:24.611823 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611797 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:39:24.611915 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611894 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:39:24.611964 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611913 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:39:24.611964 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611926 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:39:24.611964 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611953 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-glzqf\"" Oct 30 21:39:24.612097 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611915 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d8rvn\"" Oct 30 21:39:24.612097 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.611983 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:39:24.612097 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.612062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8td58\"" Oct 30 21:39:24.613402 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613386 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rvhrj\"" Oct 30 21:39:24.613494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613391 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gx2bs\"" Oct 30 21:39:24.613494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613415 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:39:24.613494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613414 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9m67w\"" Oct 30 21:39:24.613494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613414 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-csxns\"" Oct 30 21:39:24.613494 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613459 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wpbfh\"" Oct 30 21:39:24.613672 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4qw75\"" Oct 30 21:39:24.613706 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613676 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:39:24.613912 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613895 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.613983 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613900 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.613983 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613948 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.613983 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613956 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.613993 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614016 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614018 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614032 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614017 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.614125 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614057 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.614409 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.614389 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:39:24.615067 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615054 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:39:24.615141 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615081 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.615421 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615400 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:39:24.615421 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615400 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615400 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615441 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615455 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615457 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615532 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.615551 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.615540 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.621751 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.621729 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:39:24.683669 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.683669 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.683820 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.683820 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.683820 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhj6\" (UniqueName: \"kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6\") pod \"kas-connection-checker-685895c7c9-9jmb6\" (UID: \"b589e26c-f56e-426a-9c25-396f65e0488f\") " pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:24.683820 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlcg4\" (UniqueName: \"kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.683820 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.683970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.684049 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.684427 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxpz\" (UniqueName: \"kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.684427 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.684427 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9ct\" (UniqueName: \"kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.684427 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.684679 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.684727 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.684727 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.684703 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:24.684727 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8r8\" (UniqueName: \"kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.684727 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.684720 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.684774 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.1847562 +0000 UTC m=+31.057520645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.684920 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh7q\" (UniqueName: \"kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q\") pod \"kas-connection-checker-685895c7c9-kwwwz\" (UID: \"9b036611-6b37-40c6-9a5d-01e3f42d6e1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.684951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.685205 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g62x\" (UniqueName: \"kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x\") pod \"kas-connection-checker-685895c7c9-r7rws\" (UID: \"98c78665-fec7-466f-a032-51b497e7028a\") " pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5584\" (UniqueName: \"kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lr7b\" (UniqueName: \"kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b\") pod \"network-check-source-6f78d59f99-hqp48\" (UID: \"24bb5666-dc8c-400a-b112-508ea1a660de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685572 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.685650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57r2\" (UniqueName: \"kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.685957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjfq\" (UniqueName: \"kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.686396 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzpm\" (UniqueName: \"kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzztw\" (UniqueName: \"kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw\") pod \"volume-data-source-validator-58c9f5f46d-zn459\" (UID: \"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4cq\" (UniqueName: \"kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.686564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.686657 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.686671 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:24.686819 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.686708 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.186697065 +0000 UTC m=+31.059461497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:24.688409 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.688377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.688409 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.688399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.688522 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.688374 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.688522 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.688386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.701178 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.701129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.701436 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.701409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.717402 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.717380 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:24.717766 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.717561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:24.717766 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.717639 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh7q\" (UniqueName: \"kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q\") pod \"kas-connection-checker-685895c7c9-kwwwz\" (UID: \"9b036611-6b37-40c6-9a5d-01e3f42d6e1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:24.724232 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.724212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g62x\" (UniqueName: \"kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x\") pod \"kas-connection-checker-685895c7c9-r7rws\" (UID: \"98c78665-fec7-466f-a032-51b497e7028a\") " pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:24.787433 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.787433 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57r2\" (UniqueName: \"kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.787607 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjfq\" (UniqueName: \"kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.787607 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.787607 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.787607 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzpm\" (UniqueName: \"kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzztw\" (UniqueName: \"kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw\") pod \"volume-data-source-validator-58c9f5f46d-zn459\" (UID: \"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.787709 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.787717 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.787792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.287772673 +0000 UTC m=+31.160537123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.787813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls podName:fb903f58-35c6-449e-8350-3382d4494692 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.287802754 +0000 UTC m=+31.160567188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xmhg9" (UID: "fb903f58-35c6-449e-8350-3382d4494692") : secret "samples-operator-tls" not found Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.787873 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4cq\" (UniqueName: \"kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.787952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhj6\" (UniqueName: \"kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6\") pod \"kas-connection-checker-685895c7c9-9jmb6\" (UID: \"b589e26c-f56e-426a-9c25-396f65e0488f\") " pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlcg4\" (UniqueName: \"kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788065 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.788450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.788473 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.288456274 +0000 UTC m=+31.161220731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxpz\" (UniqueName: \"kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9ct\" (UniqueName: \"kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8r8\" (UniqueName: \"kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788664 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.788877 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.788856 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788934 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.788944 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls podName:dac1afb9-e804-406f-9064-8fe8e976e0c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.288929367 +0000 UTC m=+31.161693806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gfljx" (UID: "dac1afb9-e804-406f-9064-8fe8e976e0c6") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.788950 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.788964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.788986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert podName:e33c44f7-69f3-4f6e-bbfd-760ba6440f2c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.288972964 +0000 UTC m=+31.161737418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-92mlp" (UID: "e33c44f7-69f3-4f6e-bbfd-760ba6440f2c") : secret "networking-console-plugin-cert" not found Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.789016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.789046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5584\" (UniqueName: \"kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.789070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lr7b\" (UniqueName: \"kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b\") pod \"network-check-source-6f78d59f99-hqp48\" (UID: \"24bb5666-dc8c-400a-b112-508ea1a660de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.789107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.789759 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.789485 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.790568 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.790022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.790568 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.790109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.790568 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.790213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.790568 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.790332 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:39:24.790568 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:24.790385 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls podName:4763a8d1-c6f0-49eb-81e3-a6919c5f4a85 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.290371035 +0000 UTC m=+31.163135469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls") pod "dns-default-hm8lm" (UID: "4763a8d1-c6f0-49eb-81e3-a6919c5f4a85") : secret "dns-default-metrics-tls" not found Oct 30 21:39:24.790860 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.790627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.790860 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.790646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.791043 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.791020 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.791394 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.791366 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.791574 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.791551 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.792023 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.791999 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.792709 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.792684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.792792 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.792732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.793038 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.793018 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.793304 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.793288 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.799698 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.799679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:24.805293 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.805252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57r2\" (UniqueName: \"kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.806743 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.806720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:24.807372 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.807350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lr7b\" (UniqueName: \"kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b\") pod \"network-check-source-6f78d59f99-hqp48\" (UID: \"24bb5666-dc8c-400a-b112-508ea1a660de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:24.807520 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.807464 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9ct\" (UniqueName: \"kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:24.811108 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.810810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzpm\" (UniqueName: \"kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:24.811108 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.810809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5584\" (UniqueName: \"kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:24.813479 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.813430 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxpz\" (UniqueName: \"kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.813995 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.813971 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjfq\" (UniqueName: \"kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:24.814082 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.814071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4cq\" (UniqueName: \"kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:24.814138 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.814085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlcg4\" (UniqueName: \"kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.814290 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.814239 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8r8\" (UniqueName: \"kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.814521 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.814504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhj6\" (UniqueName: \"kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6\") pod \"kas-connection-checker-685895c7c9-9jmb6\" (UID: \"b589e26c-f56e-426a-9c25-396f65e0488f\") " pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:24.816905 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.816883 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzztw\" (UniqueName: \"kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw\") pod \"volume-data-source-validator-58c9f5f46d-zn459\" (UID: \"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:24.839333 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.839303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:24.859455 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.859431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:24.878370 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.877554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:24.892450 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.892248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:24.905294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.903825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:24.910475 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.910373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:24.917512 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.917489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:24.930100 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:24.929933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:25.104518 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.104441 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7rws"] Oct 30 21:39:25.107197 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.107180 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwwwz"] Oct 30 21:39:25.143031 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.142963 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jmb6"] Oct 30 21:39:25.187511 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.187441 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wh95c"] Oct 30 21:39:25.192511 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:25.192373 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b673e62_070f_4a1d_969c_1c4958789ba9.slice/crio-6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58 WatchSource:0}: Error finding container 6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58: Status 404 returned error can't find the container with id 6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58 Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.199062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.199141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199238 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199259 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199340 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.19932032 +0000 UTC m=+32.072084776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199345 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199360 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:25.199420 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.199393 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.199382525 +0000 UTC m=+32.072146985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:25.244184 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.244157 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dljwb"] Oct 30 21:39:25.246551 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:25.246527 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9829f19b_65f8_4435_a942_0c997ee268c3.slice/crio-24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f WatchSource:0}: Error finding container 24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f: Status 404 returned error can't find the container with id 24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f Oct 30 21:39:25.258477 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.258453 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459"] Oct 30 21:39:25.280691 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:25.280646 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675ec9b6_88b1_4fe3_a589_a0bcd26bfb38.slice/crio-b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37 WatchSource:0}: Error finding container b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37: Status 404 returned error can't find the container with id b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37 Oct 30 21:39:25.300923 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.300890 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48"] Oct 30 21:39:25.301674 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:25.301743 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:25.301796 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:25.301842 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:25.301842 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301815 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:39:25.301932 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:25.301932 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301852 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:25.301932 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls podName:4763a8d1-c6f0-49eb-81e3-a6919c5f4a85 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.301858952 +0000 UTC m=+32.174623387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls") pod "dns-default-hm8lm" (UID: "4763a8d1-c6f0-49eb-81e3-a6919c5f4a85") : secret "dns-default-metrics-tls" not found Oct 30 21:39:25.301932 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.301893416 +0000 UTC m=+32.174657852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:25.301932 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301918 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.301936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301949 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert podName:e33c44f7-69f3-4f6e-bbfd-760ba6440f2c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.301940252 +0000 UTC m=+32.174704711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-92mlp" (UID: "e33c44f7-69f3-4f6e-bbfd-760ba6440f2c") : secret "networking-console-plugin-cert" not found Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.301959 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.302000 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls podName:fb903f58-35c6-449e-8350-3382d4494692 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.301987582 +0000 UTC m=+32.174752016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xmhg9" (UID: "fb903f58-35c6-449e-8350-3382d4494692") : secret "samples-operator-tls" not found Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.302000 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.302024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.302016894 +0000 UTC m=+32.174781374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:25.302131 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:25.302040 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls podName:dac1afb9-e804-406f-9064-8fe8e976e0c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:26.302029056 +0000 UTC m=+32.174793493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gfljx" (UID: "dac1afb9-e804-406f-9064-8fe8e976e0c6") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:25.302490 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.302459 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p"] Oct 30 21:39:25.304134 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:25.304110 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bb5666_dc8c_400a_b112_508ea1a660de.slice/crio-5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b WatchSource:0}: Error finding container 5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b: Status 404 returned error can't find the container with id 5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b Oct 30 21:39:25.304813 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:25.304764 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc38fec3_2436_4e9e_aea8_4bf2fdab438e.slice/crio-3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db WatchSource:0}: Error finding container 3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db: Status 404 returned error can't find the container with id 3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db Oct 30 21:39:25.349934 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.349913 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw"] Oct 30 21:39:25.351009 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.350978 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkw8"] Oct 30 21:39:25.714849 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.714817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:25.715024 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.714917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:25.715024 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.714938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:25.719863 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.719836 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:39:25.720297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.720090 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:39:25.720297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.720112 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\"" Oct 30 21:39:25.720297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.720128 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\"" Oct 30 21:39:25.720297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.720156 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\"" Oct 30 21:39:25.979245 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.979173 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" event={"ID":"b589e26c-f56e-426a-9c25-396f65e0488f","Type":"ContainerStarted","Data":"91fb646646c4e883c7d4237cd59ce757a1c87c3f19757c6c216807cba65afdcc"} Oct 30 21:39:25.979245 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.979240 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" event={"ID":"b589e26c-f56e-426a-9c25-396f65e0488f","Type":"ContainerStarted","Data":"eb7d0ea37f82d56890b42835a01bb482c4d38b3aa3eb8ffa0763c29e2aee3c01"} Oct 30 21:39:25.980802 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.980744 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" event={"ID":"c6c881db-f097-4bf2-af04-3aa394bdfc44","Type":"ContainerStarted","Data":"b0e9c6e76c284c648e69d3a47ad0e3768220b5f1d65d1b2bab4650e306226393"} Oct 30 21:39:25.982648 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.982593 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" event={"ID":"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38","Type":"ContainerStarted","Data":"b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37"} Oct 30 21:39:25.984971 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.984947 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wh95c" event={"ID":"6b673e62-070f-4a1d-969c-1c4958789ba9","Type":"ContainerStarted","Data":"6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58"} Oct 30 21:39:25.986815 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.986762 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-r7rws" event={"ID":"98c78665-fec7-466f-a032-51b497e7028a","Type":"ContainerStarted","Data":"1cbe437ef569c95c226e7bf88d2b259a6bf6c05d75b67f3ca3d8e6221a74e07a"} Oct 30 21:39:25.986815 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.986810 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-r7rws" event={"ID":"98c78665-fec7-466f-a032-51b497e7028a","Type":"ContainerStarted","Data":"8cfe55a74bf5bf37209bafe91eccd2ce08f2a6b7fad4e579ffd2cb59a10ca6e7"} Oct 30 21:39:25.988717 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.988692 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" event={"ID":"9b036611-6b37-40c6-9a5d-01e3f42d6e1e","Type":"ContainerStarted","Data":"edb517ca21a778553520b18ad8f9ddb4122f1650ee7e63602811cbb6b5c1dd50"} Oct 30 21:39:25.988829 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.988735 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" event={"ID":"9b036611-6b37-40c6-9a5d-01e3f42d6e1e","Type":"ContainerStarted","Data":"ca1d6c132313e6dac8557a462d53ded2d50902cea7ef508f1d3a66bff60f6a1a"} Oct 30 21:39:25.991372 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.991347 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxkw8" event={"ID":"119335ed-3944-46c6-aacd-e63aff054eb8","Type":"ContainerStarted","Data":"8c94c2d701ae91309d9d95657d1827c76e43fc8ec811c5609af0a6cd50006490"} Oct 30 21:39:25.994446 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.994322 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" event={"ID":"24bb5666-dc8c-400a-b112-508ea1a660de","Type":"ContainerStarted","Data":"5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b"} Oct 30 21:39:25.998749 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.998724 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" event={"ID":"cc38fec3-2436-4e9e-aea8-4bf2fdab438e","Type":"ContainerStarted","Data":"3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db"} Oct 30 21:39:25.999104 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:25.999057 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" podStartSLOduration=391.999040827 podStartE2EDuration="6m31.999040827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:25.997545044 +0000 UTC m=+31.870309499" watchObservedRunningTime="2026-10-30 21:39:25.999040827 +0000 UTC m=+31.871805283" Oct 30 21:39:26.001672 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.001648 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" event={"ID":"9829f19b-65f8-4435-a942-0c997ee268c3","Type":"ContainerStarted","Data":"24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f"} Oct 30 21:39:26.017671 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.017632 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-r7rws" podStartSLOduration=392.017623063 podStartE2EDuration="6m32.017623063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.016509511 +0000 UTC m=+31.889273966" watchObservedRunningTime="2026-10-30 21:39:26.017623063 +0000 UTC m=+31.890387511" Oct 30 21:39:26.037308 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.037250 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" podStartSLOduration=392.037236165 podStartE2EDuration="6m32.037236165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.036789428 +0000 UTC m=+31.909553884" watchObservedRunningTime="2026-10-30 21:39:26.037236165 +0000 UTC m=+31.910000622" Oct 30 21:39:26.209358 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.209141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:26.209358 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.209249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:26.209525 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.209422 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:26.209525 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.209440 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:26.209598 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.209576 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:26.209598 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.209588 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:26.209669 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.209633 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.20961384 +0000 UTC m=+34.082378291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:26.210286 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.210148 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.210118003 +0000 UTC m=+34.082882450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:26.310734 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:26.310734 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:26.310926 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:26.310926 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:26.310926 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:26.311025 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:26.310936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:26.311100 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311075 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:39:26.311212 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311144 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls podName:fb903f58-35c6-449e-8350-3382d4494692 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311126867 +0000 UTC m=+34.183891313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xmhg9" (UID: "fb903f58-35c6-449e-8350-3382d4494692") : secret "samples-operator-tls" not found Oct 30 21:39:26.311587 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311557 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311540209 +0000 UTC m=+34.184304654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:26.311740 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311642 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:39:26.311740 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311686 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert podName:e33c44f7-69f3-4f6e-bbfd-760ba6440f2c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311673751 +0000 UTC m=+34.184438183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-92mlp" (UID: "e33c44f7-69f3-4f6e-bbfd-760ba6440f2c") : secret "networking-console-plugin-cert" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311749 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311803 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls podName:dac1afb9-e804-406f-9064-8fe8e976e0c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311773467 +0000 UTC m=+34.184538112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gfljx" (UID: "dac1afb9-e804-406f-9064-8fe8e976e0c6") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311867 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311912 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls podName:4763a8d1-c6f0-49eb-81e3-a6919c5f4a85 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311899761 +0000 UTC m=+34.184664210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls") pod "dns-default-hm8lm" (UID: "4763a8d1-c6f0-49eb-81e3-a6919c5f4a85") : secret "dns-default-metrics-tls" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.311975 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:26.312008 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:26.312009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:28.311998059 +0000 UTC m=+34.184762504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:28.018713 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.017887 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" event={"ID":"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38","Type":"ContainerStarted","Data":"d881b09b080ab7307a72b4d40d9d00ecaf1bce8e9d18e1307306abdf96878bb6"} Oct 30 21:39:28.040691 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.039696 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" podStartSLOduration=358.850586674 podStartE2EDuration="6m1.039660864s" totalImagesPullingTime="2.18907419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:28.037390338 +0000 UTC m=+33.910154793" watchObservedRunningTime="2026-10-30 21:39:28.039660864 +0000 UTC m=+33.912425319" Oct 30 21:39:28.234174 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.234132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.234356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.234515 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.234533 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.234597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.234576889 +0000 UTC m=+38.107341323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.235052 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.235069 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:28.235146 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.235115 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.235100699 +0000 UTC m=+38.107865133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.334808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.334884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.334919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.334966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.335017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.335072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.335107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335245 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls podName:fb903f58-35c6-449e-8350-3382d4494692 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.335311236 +0000 UTC m=+38.208075688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xmhg9" (UID: "fb903f58-35c6-449e-8350-3382d4494692") : secret "samples-operator-tls" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335743 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.335725508 +0000 UTC m=+38.208489955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335819 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert podName:e33c44f7-69f3-4f6e-bbfd-760ba6440f2c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.335844267 +0000 UTC m=+38.208608703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-92mlp" (UID: "e33c44f7-69f3-4f6e-bbfd-760ba6440f2c") : secret "networking-console-plugin-cert" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335922 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.335952 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls podName:dac1afb9-e804-406f-9064-8fe8e976e0c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.33594212 +0000 UTC m=+38.208706555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gfljx" (UID: "dac1afb9-e804-406f-9064-8fe8e976e0c6") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:28.336294 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336002 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:39:28.337032 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336031 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls podName:4763a8d1-c6f0-49eb-81e3-a6919c5f4a85 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.336020735 +0000 UTC m=+38.208785169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls") pod "dns-default-hm8lm" (UID: "4763a8d1-c6f0-49eb-81e3-a6919c5f4a85") : secret "dns-default-metrics-tls" not found Oct 30 21:39:28.337032 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336086 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:39:28.337032 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:00.336103439 +0000 UTC m=+66.208867876 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : secret "metrics-daemon-secret" not found Oct 30 21:39:28.337032 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336165 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:28.337032 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:28.336193 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.336182727 +0000 UTC m=+38.208947174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:28.436617 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.435970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:28.440185 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.440154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:28.451332 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.451123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:28.536664 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.536630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:28.560960 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.560922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:28.800778 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.800564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:28.803645 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:28.803611 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd669"] Oct 30 21:39:28.815311 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:28.813583 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a37659_0e94_43e8_adb2_94ec2dfe09a1.slice/crio-4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38 WatchSource:0}: Error finding container 4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38: Status 404 returned error can't find the container with id 4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38 Oct 30 21:39:29.023003 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:29.022928 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd669" event={"ID":"92a37659-0e94-43e8-adb2-94ec2dfe09a1","Type":"ContainerStarted","Data":"4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38"} Oct 30 21:39:29.135302 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:29.135255 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dr5t5"] Oct 30 21:39:29.171040 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:29.171005 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc8ab91_c338_4320_b006_574a196d2333.slice/crio-95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28 WatchSource:0}: Error finding container 95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28: Status 404 returned error can't find the container with id 95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28 Oct 30 21:39:30.028784 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:30.028691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dr5t5" event={"ID":"0dc8ab91-c338-4320-b006-574a196d2333","Type":"ContainerStarted","Data":"95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28"} Oct 30 21:39:30.030711 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:30.030682 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/0.log" Oct 30 21:39:30.030832 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:30.030724 2576 generic.go:356] "Generic (PLEG): container finished" podID="9829f19b-65f8-4435-a942-0c997ee268c3" containerID="4cbf0f0f412525ac87b9c32e0753297d0ae51849608cc8db3d726fc39f2478c1" exitCode=255 Oct 30 21:39:30.030832 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:30.030772 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" event={"ID":"9829f19b-65f8-4435-a942-0c997ee268c3","Type":"ContainerDied","Data":"4cbf0f0f412525ac87b9c32e0753297d0ae51849608cc8db3d726fc39f2478c1"} Oct 30 21:39:30.031463 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:30.031111 2576 scope.go:122] "RemoveContainer" containerID="4cbf0f0f412525ac87b9c32e0753297d0ae51849608cc8db3d726fc39f2478c1" Oct 30 21:39:31.039200 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.038473 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" event={"ID":"cc38fec3-2436-4e9e-aea8-4bf2fdab438e","Type":"ContainerStarted","Data":"e4afcd323be01722656e722fadbf2984aed59dbd602abbe37b302de9f669a686"} Oct 30 21:39:31.046220 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.046139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.047717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/0.log" Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.047753 2576 generic.go:356] "Generic (PLEG): container finished" podID="9829f19b-65f8-4435-a942-0c997ee268c3" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" exitCode=255 Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.047830 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" event={"ID":"9829f19b-65f8-4435-a942-0c997ee268c3","Type":"ContainerDied","Data":"dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f"} Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.047877 2576 scope.go:122] "RemoveContainer" containerID="4cbf0f0f412525ac87b9c32e0753297d0ae51849608cc8db3d726fc39f2478c1" Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.048260 2576 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:31.049495 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.048499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:31.056839 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.056363 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" event={"ID":"c6c881db-f097-4bf2-af04-3aa394bdfc44","Type":"ContainerStarted","Data":"d22dbee888f12bfa6e9b845b3c0eaf45b5f3d5cea4d57bc8793297c648bf7497"} Oct 30 21:39:31.063838 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.063769 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wh95c" event={"ID":"6b673e62-070f-4a1d-969c-1c4958789ba9","Type":"ContainerStarted","Data":"849c568fa1be6f498d38f0ac869ea137028b70f2c5fa05f71d15b74437b6ab46"} Oct 30 21:39:31.065496 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.065476 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxkw8" event={"ID":"119335ed-3944-46c6-aacd-e63aff054eb8","Type":"ContainerStarted","Data":"d258ef55942baad296a12dc9e8d32b09bab6545fd4434dbdfe85b7eb7e9a73ed"} Oct 30 21:39:31.107390 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.106865 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" podStartSLOduration=358.987578012 podStartE2EDuration="6m4.106842603s" totalImagesPullingTime="5.119264591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:31.069026078 +0000 UTC m=+36.941790532" watchObservedRunningTime="2026-10-30 21:39:31.106842603 +0000 UTC m=+36.979607058" Oct 30 21:39:31.131513 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.129115 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n"] Oct 30 21:39:31.157523 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.157487 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht"] Oct 30 21:39:31.157698 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.157592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:31.172777 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.172739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:31.172777 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.172776 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:31.173261 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.173218 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jntk6\"" Oct 30 21:39:31.173366 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.173292 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:39:31.177721 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.177635 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n"] Oct 30 21:39:31.178710 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.178629 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wh95c" podStartSLOduration=358.967812652 podStartE2EDuration="6m4.178614172s" totalImagesPullingTime="5.21080152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:31.159804443 +0000 UTC m=+37.032568907" watchObservedRunningTime="2026-10-30 21:39:31.178614172 +0000 UTC m=+37.051378627" Oct 30 21:39:31.196832 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.196700 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" podStartSLOduration=359.142487685 podStartE2EDuration="6m4.196680489s" totalImagesPullingTime="5.054192804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:31.193999563 +0000 UTC m=+37.066764017" watchObservedRunningTime="2026-10-30 21:39:31.196680489 +0000 UTC m=+37.069444944" Oct 30 21:39:31.197611 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.197586 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht"] Oct 30 21:39:31.226314 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.226243 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fxkw8" podStartSLOduration=1.7284203009999999 podStartE2EDuration="7.226227968s" totalImagesPullingTime="5.497807667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:31.225990802 +0000 UTC m=+37.098755253" watchObservedRunningTime="2026-10-30 21:39:31.226227968 +0000 UTC m=+37.098992422" Oct 30 21:39:31.260394 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.260362 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:39:31.272625 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.268650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.272625 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.268507 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jqzzl"] Oct 30 21:39:31.272625 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.271832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7cs\" (UniqueName: \"kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs\") pod \"migrator-66df5d496b-rlsht\" (UID: \"d0e15f10-7732-42c8-a07e-dddaf048f664\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:31.281326 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.273559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fld\" (UniqueName: \"kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld\") pod \"migrator-66df5d496b-5h44n\" (UID: \"e7174f1b-4841-4952-8f04-331347493639\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:31.282456 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.282434 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:39:31.285319 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.282691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.289565 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.289542 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqzzl"] Oct 30 21:39:31.294997 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.294977 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:39:31.295159 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.295145 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45g4h\"" Oct 30 21:39:31.368240 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.368143 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:39:31.382692 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.382863 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.382863 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.382863 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7cs\" (UniqueName: \"kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs\") pod \"migrator-66df5d496b-rlsht\" (UID: \"d0e15f10-7732-42c8-a07e-dddaf048f664\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:31.383027 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.383027 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fld\" (UniqueName: \"kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld\") pod \"migrator-66df5d496b-5h44n\" (UID: \"e7174f1b-4841-4952-8f04-331347493639\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:31.383027 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.383027 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2br88\" (UniqueName: \"kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.383027 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.382999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.383251 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.383032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.383251 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.383060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.407712 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.407660 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7cs\" (UniqueName: \"kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs\") pod \"migrator-66df5d496b-rlsht\" (UID: \"d0e15f10-7732-42c8-a07e-dddaf048f664\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:31.407853 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.407730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fld\" (UniqueName: \"kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld\") pod \"migrator-66df5d496b-5h44n\" (UID: \"e7174f1b-4841-4952-8f04-331347493639\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:31.484362 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.484542 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.484542 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.484653 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.484653 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.484751 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br88\" (UniqueName: \"kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.484751 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.484751 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.484898 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.484762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.485244 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.485222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.485367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.485461 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.485503 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:31.985489726 +0000 UTC m=+37.858254159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.486168 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.486232 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:31.986214539 +0000 UTC m=+37.858978986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.486722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.487095 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.487031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.487719 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.487697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.494221 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.494195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:31.502213 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.502181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.503824 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.503776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br88\" (UniqueName: \"kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.505539 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.505518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:31.601336 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.600609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:31.617984 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.617914 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tkt5h"] Oct 30 21:39:31.643221 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.643193 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tkt5h"] Oct 30 21:39:31.643360 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.643339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.648556 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.648533 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:39:31.649285 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.648684 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xqfqq\"" Oct 30 21:39:31.649285 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.648737 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:39:31.649285 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.648829 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:39:31.649285 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.648902 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:39:31.687832 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.687803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.687987 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.687844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.688073 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.688047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v76nk\" (UniqueName: \"kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.788828 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.788805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v76nk\" (UniqueName: \"kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.789043 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.789011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.789566 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.789042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.791209 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.791176 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.800654 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.800591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.803898 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.803874 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v76nk\" (UniqueName: \"kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.818965 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.818935 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n"] Oct 30 21:39:31.823171 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.823151 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht"] Oct 30 21:39:31.831898 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:31.831441 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7174f1b_4841_4952_8f04_331347493639.slice/crio-9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211 WatchSource:0}: Error finding container 9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211: Status 404 returned error can't find the container with id 9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211 Oct 30 21:39:31.832588 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:31.832549 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e15f10_7732_42c8_a07e_dddaf048f664.slice/crio-841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a WatchSource:0}: Error finding container 841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a: Status 404 returned error can't find the container with id 841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a Oct 30 21:39:31.985202 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.985116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:31.991584 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.991544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.991709 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:31.991685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:31.991709 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.991696 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:31.991786 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.991775 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.99175089 +0000 UTC m=+38.864515329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:31.991871 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.991858 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:31.991920 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:31.991911 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.991899906 +0000 UTC m=+38.864664339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:32.074227 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.074004 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:39:32.101295 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.099819 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" event={"ID":"24bb5666-dc8c-400a-b112-508ea1a660de","Type":"ContainerStarted","Data":"a70cce9106512ae59d40afac8a4a6bae75ad476457a78b841b5dad3a6ca76b66"} Oct 30 21:39:32.103286 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.103242 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" event={"ID":"d0e15f10-7732-42c8-a07e-dddaf048f664","Type":"ContainerStarted","Data":"841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a"} Oct 30 21:39:32.105429 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.105349 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" event={"ID":"e7174f1b-4841-4952-8f04-331347493639","Type":"ContainerStarted","Data":"9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211"} Oct 30 21:39:32.113297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.111834 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:32.113297 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.112551 2576 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:32.113297 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.112666 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:32.120789 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.120745 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" podStartSLOduration=358.640844556 podStartE2EDuration="6m5.12072846s" totalImagesPullingTime="6.479883904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:32.118673219 +0000 UTC m=+37.991437675" watchObservedRunningTime="2026-10-30 21:39:32.12072846 +0000 UTC m=+37.993492916" Oct 30 21:39:32.293995 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.293730 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tkt5h"] Oct 30 21:39:32.295959 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.295928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:32.296152 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.296041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:32.296152 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296115 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:32.296152 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296133 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:32.296348 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296181 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.296165556 +0000 UTC m=+46.168930008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:32.296348 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296216 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:32.296348 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296228 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:32.296348 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.296285 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.296253549 +0000 UTC m=+46.169017984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:32.296682 ip-10-0-133-193 kubenswrapper[2576]: W1030 21:39:32.296648 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194d4b78_7c26_401b_858b_07730092cc53.slice/crio-86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f WatchSource:0}: Error finding container 86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f: Status 404 returned error can't find the container with id 86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:32.397862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398041 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398103 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls podName:4763a8d1-c6f0-49eb-81e3-a6919c5f4a85 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398084904 +0000 UTC m=+46.270849351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls") pod "dns-default-hm8lm" (UID: "4763a8d1-c6f0-49eb-81e3-a6919c5f4a85") : secret "dns-default-metrics-tls" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398406 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398386709 +0000 UTC m=+46.271151142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398493 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398510 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398512 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398529 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398517279 +0000 UTC m=+46.271281716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398532 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:39:32.398659 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398560 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert podName:e33c44f7-69f3-4f6e-bbfd-760ba6440f2c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398543606 +0000 UTC m=+46.271308044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-92mlp" (UID: "e33c44f7-69f3-4f6e-bbfd-760ba6440f2c") : secret "networking-console-plugin-cert" not found Oct 30 21:39:32.399506 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398578 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls podName:dac1afb9-e804-406f-9064-8fe8e976e0c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398568314 +0000 UTC m=+46.271332752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gfljx" (UID: "dac1afb9-e804-406f-9064-8fe8e976e0c6") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:39:32.399506 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:32.398597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls podName:fb903f58-35c6-449e-8350-3382d4494692 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.398588586 +0000 UTC m=+46.271353023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xmhg9" (UID: "fb903f58-35c6-449e-8350-3382d4494692") : secret "samples-operator-tls" not found Oct 30 21:39:33.009836 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.009798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:33.010012 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.009943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:33.010012 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:33.009965 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:33.010124 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:33.010042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:35.010021246 +0000 UTC m=+40.882785694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:33.010124 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:33.010080 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:33.010318 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:33.010130 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:35.010115936 +0000 UTC m=+40.882880377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:33.119359 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.119325 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" event={"ID":"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191","Type":"ContainerStarted","Data":"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3"} Oct 30 21:39:33.119807 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.119381 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" event={"ID":"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191","Type":"ContainerStarted","Data":"cafbd56f7d5a6e255efd7cad05bc4ce91951415b5d3a6a5e75dc8aead9accb9b"} Oct 30 21:39:33.122001 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.121977 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" event={"ID":"194d4b78-7c26-401b-858b-07730092cc53","Type":"ContainerStarted","Data":"66663d6dfd3373f773b967987c62e712814615e03eb91c26d0c7b757c64cbe35"} Oct 30 21:39:33.122156 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.122017 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" event={"ID":"194d4b78-7c26-401b-858b-07730092cc53","Type":"ContainerStarted","Data":"86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f"} Oct 30 21:39:33.139732 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.139695 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" podStartSLOduration=2.139683313 podStartE2EDuration="2.139683313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:33.138171399 +0000 UTC m=+39.010935852" watchObservedRunningTime="2026-10-30 21:39:33.139683313 +0000 UTC m=+39.012447766" Oct 30 21:39:33.165606 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:33.165566 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" podStartSLOduration=2.165553628 podStartE2EDuration="2.165553628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:33.164209407 +0000 UTC m=+39.036973866" watchObservedRunningTime="2026-10-30 21:39:33.165553628 +0000 UTC m=+39.038318082" Oct 30 21:39:34.128876 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.128833 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dr5t5" event={"ID":"0dc8ab91-c338-4320-b006-574a196d2333","Type":"ContainerStarted","Data":"6238521896a75d6d5cd63befd9af98ce2b74052bb488cd77406e6e05abd2d730"} Oct 30 21:39:34.148641 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.148598 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dr5t5" podStartSLOduration=35.009433757 podStartE2EDuration="39.14858414s" totalImagesPullingTime="4.139150383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.147518279 +0000 UTC m=+40.020282777" watchObservedRunningTime="2026-10-30 21:39:34.14858414 +0000 UTC m=+40.021348594" Oct 30 21:39:34.400853 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.400819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h7xwb_5c533f37-a794-4b53-8779-86fbbb312bf6/dns-node-resolver/0.log" Oct 30 21:39:34.893327 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.893290 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:34.893518 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.893361 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:34.893786 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:34.893760 2576 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:34.894018 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:34.893998 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:35.036650 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.036605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:35.036835 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.036775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:35.036835 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:35.036822 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:35.036956 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:35.036895 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:39.036874396 +0000 UTC m=+44.909638843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:35.037107 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:35.037090 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:35.037164 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:35.037147 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:39.037130107 +0000 UTC m=+44.909894554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:35.134756 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.134716 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" event={"ID":"d0e15f10-7732-42c8-a07e-dddaf048f664","Type":"ContainerStarted","Data":"1f815dc944383b6d02ec4dc21044140d1bba68c40ec56c710696194df3189e4b"} Oct 30 21:39:35.136481 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.136442 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" event={"ID":"e7174f1b-4841-4952-8f04-331347493639","Type":"ContainerStarted","Data":"76fd6f7878366f7f7cce3b87620d5b8a9dd86369e9d0745c86ed5df08a16c322"} Oct 30 21:39:35.136774 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.136757 2576 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:35.136874 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.136810 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:35.136964 ip-10-0-133-193 kubenswrapper[2576]: E1030 21:39:35.136943 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:35.154921 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.154838 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" podStartSLOduration=1.510185726 podStartE2EDuration="4.154825817s" totalImagesPullingTime="2.644640091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:35.15240002 +0000 UTC m=+41.025164471" watchObservedRunningTime="2026-10-30 21:39:35.154825817 +0000 UTC m=+41.027590271" Oct 30 21:39:35.599365 ip-10-0-133-193 kubenswrapper[2576]: I1030 21:39:35.599287 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgsq6_d6509794-be77-4085-aeec-92819c3c9041/node-ca/0.log" Oct 30 21:39:35.991707 ip-10-0-133-193 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:39:36.004735 ip-10-0-133-193 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:39:36.004902 ip-10-0-133-193 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:39:36.004951 ip-10-0-133-193 systemd[1]: kubelet.service: Consumed 3.169s CPU time. Oct 30 21:39:36.018861 ip-10-0-133-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:39:36.090475 ip-10-0-133-193 kubenswrapper[5036]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:36.090475 ip-10-0-133-193 kubenswrapper[5036]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:39:36.090475 ip-10-0-133-193 kubenswrapper[5036]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:36.090475 ip-10-0-133-193 kubenswrapper[5036]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093662 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093746 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093750 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093753 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093756 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.093750 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093759 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093762 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093765 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093767 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093770 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093773 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093775 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093778 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093781 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093784 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093787 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093790 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093792 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093795 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093798 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093800 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093803 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093806 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093808 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093811 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.093989 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093814 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093817 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093820 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093822 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093825 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093827 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093830 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093839 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093842 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093845 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093848 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093850 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093853 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093855 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093858 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093860 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093863 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093866 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093868 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093871 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.094527 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093873 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093876 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093878 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093881 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093884 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093886 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093889 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093891 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093894 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093897 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093899 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093902 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093905 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093908 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093911 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093914 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093916 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093919 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093922 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093925 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.095006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093927 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093930 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093934 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093938 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093942 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093945 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093948 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093951 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093954 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093957 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093959 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093962 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093965 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093967 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093970 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093973 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093976 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093978 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093982 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.095540 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093985 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093987 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093990 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093993 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.093997 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094002 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094005 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094008 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094012 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094014 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094017 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094020 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094022 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094025 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094028 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094032 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094040 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094042 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094045 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094048 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.096005 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094050 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094053 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094816 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094823 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094826 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094830 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094834 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094836 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094839 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094842 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094844 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094847 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094850 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094853 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094856 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094858 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094862 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094865 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094867 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094870 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.096518 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094872 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094875 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094878 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094880 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094883 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094885 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094888 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094891 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094896 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094898 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094901 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094904 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094907 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094909 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094912 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094915 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094918 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094920 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094923 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094925 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.097030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094928 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094931 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094933 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094936 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094938 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094942 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094944 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094947 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094950 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094952 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094955 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094957 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094960 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094962 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094965 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094967 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094970 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094973 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094975 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.097543 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094978 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094982 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094985 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094988 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094990 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094993 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094996 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.094998 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095001 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095004 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095007 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095010 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095013 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095016 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095019 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095021 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095024 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095027 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095029 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095032 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.098074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095035 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095037 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095040 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095042 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095045 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095047 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095050 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095054 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095058 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095061 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095064 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095067 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095069 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095073 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095075 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095078 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095080 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095083 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095086 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.098608 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095089 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095091 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095094 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095096 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095099 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095101 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095104 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095107 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095111 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095115 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095612 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095618 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095622 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095625 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095628 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095631 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095633 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095636 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095639 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.099074 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095643 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095679 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095685 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095689 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095693 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095696 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095699 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095703 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095706 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095708 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095711 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095714 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095716 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095719 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095722 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095724 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095727 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095729 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095732 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.099559 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095735 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095737 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095739 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095742 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095744 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095747 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095750 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095753 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095755 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095758 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095761 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095763 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095766 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095768 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095771 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095774 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095777 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095779 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095781 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095784 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.100013 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095788 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095791 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095794 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095796 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095799 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095801 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095804 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095807 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095809 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095813 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095816 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095819 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095822 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095824 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095827 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095829 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095832 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095834 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095837 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.100528 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095839 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095843 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095845 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095848 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095850 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095853 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095855 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095858 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095861 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095864 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095866 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095869 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095872 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095876 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095878 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095881 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095884 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095886 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095889 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095891 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.100982 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095894 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095896 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095899 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095901 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095904 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095906 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095909 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095912 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095914 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095917 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095920 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095922 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095925 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095928 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095931 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095934 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095936 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095939 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.095942 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096037 5036 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096051 5036 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:39:36.101549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096057 5036 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096062 5036 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096066 5036 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096069 5036 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096074 5036 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096078 5036 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096081 5036 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096084 5036 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096088 5036 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096091 5036 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096094 5036 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096097 5036 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096100 5036 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096102 5036 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096105 5036 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096108 5036 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096112 5036 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096114 5036 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096117 5036 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096120 5036 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096124 5036 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096127 5036 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096130 5036 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:39:36.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096133 5036 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096136 5036 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096139 5036 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096143 5036 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096146 5036 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096149 5036 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096151 5036 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096155 5036 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096158 5036 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096161 5036 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096164 5036 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096167 5036 flags.go:64] FLAG: --enable-server="true" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096170 5036 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096174 5036 flags.go:64] FLAG: --event-burst="100" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096177 5036 flags.go:64] FLAG: --event-qps="50" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096180 5036 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096183 5036 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096186 5036 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096190 5036 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096193 5036 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096196 5036 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096199 5036 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096202 5036 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096205 5036 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096208 5036 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:39:36.102643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096211 5036 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096213 5036 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096216 5036 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096219 5036 flags.go:64] FLAG: --feature-gates="" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096223 5036 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096226 5036 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096229 5036 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096232 5036 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096235 5036 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096238 5036 flags.go:64] FLAG: --help="false" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096241 5036 flags.go:64] FLAG: --hostname-override="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096245 5036 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096250 5036 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096253 5036 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096256 5036 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096260 5036 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096281 5036 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096285 5036 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096288 5036 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096291 5036 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096294 5036 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096297 5036 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096300 5036 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096303 5036 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:39:36.103290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096306 5036 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096310 5036 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096312 5036 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096315 5036 flags.go:64] FLAG: --lock-file="" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096318 5036 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096321 5036 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096324 5036 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096329 5036 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096332 5036 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096335 5036 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096338 5036 flags.go:64] FLAG: --logging-format="text" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096341 5036 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096344 5036 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096347 5036 flags.go:64] FLAG: --manifest-url="" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096350 5036 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096354 5036 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096357 5036 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096361 5036 flags.go:64] FLAG: --max-pods="110" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096364 5036 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096367 5036 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096372 5036 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096375 5036 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096378 5036 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096380 5036 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096383 5036 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:39:36.103881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096399 5036 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096402 5036 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096405 5036 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096408 5036 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096411 5036 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096414 5036 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096417 5036 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096420 5036 flags.go:64] FLAG: --port="10250" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096423 5036 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096425 5036 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c7b723c4fdace55b" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096428 5036 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096431 5036 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096434 5036 flags.go:64] FLAG: --register-node="true" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096437 5036 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096441 5036 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096444 5036 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096447 5036 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096450 5036 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096454 5036 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096456 5036 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096459 5036 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096463 5036 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096465 5036 flags.go:64] FLAG: --runonce="false" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096468 5036 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096471 5036 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096474 5036 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:39:36.104509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096477 5036 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096481 5036 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096484 5036 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096487 5036 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096490 5036 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096493 5036 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096498 5036 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096500 5036 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096503 5036 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096506 5036 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096509 5036 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096512 5036 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096517 5036 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096520 5036 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096523 5036 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096527 5036 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096530 5036 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096533 5036 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096536 5036 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096539 5036 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096542 5036 flags.go:64] FLAG: --v="2" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096546 5036 flags.go:64] FLAG: --version="false" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096550 5036 flags.go:64] FLAG: --vmodule="" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096554 5036 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.096557 5036 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:39:36.105126 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096685 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096689 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096692 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096694 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096697 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096699 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096702 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096705 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096709 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096712 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096715 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096717 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096720 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096724 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096726 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096729 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096732 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096735 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096737 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096740 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.105754 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096743 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096745 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096747 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096750 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096753 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096755 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096758 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096761 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096763 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096766 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096768 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096771 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096773 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096776 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096779 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096781 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096783 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096786 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096788 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096791 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.106281 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096795 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096797 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096800 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096803 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096806 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096810 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096812 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096815 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096817 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096820 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096823 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096826 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096828 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096831 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096834 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096836 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096839 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096841 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096844 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096846 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.106827 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096849 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096852 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096854 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096857 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096859 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096862 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096866 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096870 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096874 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096877 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096880 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096883 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096887 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096889 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096892 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096895 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096905 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096909 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.107358 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096911 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096914 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096916 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096919 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096922 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096924 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096927 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096929 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096932 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096935 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096937 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096940 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096942 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096945 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096947 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096950 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096952 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096955 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096958 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.107937 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096960 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096962 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096965 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096968 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096970 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096973 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096976 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096980 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.096983 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.097009 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097048 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097053 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097060 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097065 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097068 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097071 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.108502 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097074 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097076 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097079 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097081 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097085 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097089 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097093 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097097 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097101 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097105 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097108 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097111 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097114 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097116 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097119 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097121 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097124 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097127 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097129 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097132 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.108972 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097134 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097137 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097139 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097143 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097146 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097149 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097151 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097154 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097156 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097161 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097164 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097166 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097169 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097171 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097174 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097177 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097179 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097182 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097184 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097187 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.109479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097189 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097192 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097194 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097197 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097199 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097202 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097204 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097207 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097209 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097212 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097214 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097217 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097220 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097222 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097224 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097228 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097231 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097233 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097236 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097251 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.110080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097254 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097259 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097280 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097284 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097286 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097290 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097293 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097295 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097298 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097301 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097304 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097306 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097309 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097311 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097314 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097318 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097323 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097326 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097330 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.110781 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097333 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097335 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097338 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097340 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097343 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097345 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097348 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097350 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097354 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097357 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097359 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097363 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097366 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097368 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097374 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097377 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097380 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097383 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097385 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.111290 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097388 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.097390 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.101385 5036 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.101400 5036 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.101419 5036 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.101424 5036 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101478 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101484 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101488 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101491 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101495 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101498 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101500 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101503 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101506 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101508 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101511 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101513 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101516 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101518 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.111758 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101521 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101524 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101526 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101529 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101531 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101534 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101536 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101539 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101542 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101546 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101548 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101551 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101554 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101557 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101560 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101563 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101566 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101569 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101572 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.112236 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101574 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101577 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101579 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101582 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101584 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101587 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101590 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101593 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101595 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101598 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101601 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101603 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101606 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101609 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101611 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101614 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101616 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101619 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101621 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101624 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.112890 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101626 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101629 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101632 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101634 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101637 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101640 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101643 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101646 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101649 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101651 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101654 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101656 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101659 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101661 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101664 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101666 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101669 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101671 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101674 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101677 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.113435 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101679 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101682 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101685 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101688 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101690 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101693 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101696 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101699 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101701 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101704 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101706 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101709 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101712 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101714 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101716 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101719 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101722 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101725 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101727 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.113930 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101731 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101734 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101737 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101740 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101743 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101746 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101748 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101751 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101753 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101756 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101758 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101762 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101766 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101769 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.101796 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:36.114406 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101926 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101931 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101934 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101937 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101940 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101942 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101944 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101947 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101950 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101952 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101955 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101958 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101960 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101963 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101965 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101969 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101971 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101974 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101976 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101979 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.114780 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101981 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101984 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101987 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101989 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101992 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101995 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.101997 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102000 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102002 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102005 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102007 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102009 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102012 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102015 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102018 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102020 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102023 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102025 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102028 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102030 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.115422 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102032 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102035 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102037 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102040 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102042 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102045 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102047 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102050 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102053 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102055 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102058 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102061 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102064 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102067 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102070 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102072 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102075 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102078 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102080 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102083 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.115953 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102085 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102088 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102091 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102093 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102095 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102098 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102102 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102106 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102109 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102111 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102114 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102116 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102119 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102121 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102124 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102126 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102128 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102131 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102133 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.116472 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102136 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102138 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102141 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102144 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102146 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102149 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102151 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102155 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102158 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102162 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102165 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102168 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102170 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102173 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102175 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102178 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102180 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102183 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102185 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.116964 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102188 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102190 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102193 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102195 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102198 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102200 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102203 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102205 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.102227 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102259 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102281 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102285 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102288 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102290 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102293 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:36.117441 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102295 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102299 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102302 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102305 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102307 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102310 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102313 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102315 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102318 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102320 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102323 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102326 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102328 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102331 5036 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102333 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102336 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102338 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102341 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102343 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:36.117800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102346 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102348 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102351 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102353 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102356 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102358 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102361 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102364 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102366 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102369 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102371 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102373 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102376 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102379 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102384 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102388 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102391 5036 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102394 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102396 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102403 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:36.118457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102405 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102408 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102411 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102413 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102416 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102419 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102421 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102424 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102427 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102429 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102432 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102434 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102437 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102441 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102445 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102447 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102450 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102453 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102456 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102458 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:36.118933 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102461 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102463 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102466 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102468 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102471 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102473 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102476 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102478 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102482 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102485 5036 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102487 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102491 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102493 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102496 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102499 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102501 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102504 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102506 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102509 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:36.119445 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102512 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102514 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102517 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102519 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102521 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102524 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102527 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102530 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102532 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102535 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102537 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102540 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102542 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102545 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102547 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102550 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102552 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102555 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102560 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102563 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:36.119895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102565 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:36.102568 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.102617 5036 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.103596 5036 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.103656 5036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.103839 5036 server.go:1014] "Starting client certificate rotation" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.103974 5036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.104031 5036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:33:54 +0000 UTC" deadline="2028-06-26 08:18:36.401708776 +0000 UTC" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.104062 5036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14506h39m0.297648236s" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.104349 5036 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.105415 5036 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.108153 5036 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.109856 5036 log.go:25] "Validated CRI v1 image API" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.110531 5036 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:39:36.120423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.112482 5036 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e7036303-b278-4b36-890d-499b248ed4b4:/dev/nvme0n1p4 f1e73ae8-eef3-44eb-a528-72dadcc58f68:/dev/nvme0n1p3] Oct 30 21:39:36.120808 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.112503 5036 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c94c2d701ae91309d9d95657d1827c76e43fc8ec811c5609af0a6cd50006490/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c94c2d701ae91309d9d95657d1827c76e43fc8ec811c5609af0a6cd50006490/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cfe55a74bf5bf37209bafe91eccd2ce08f2a6b7fad4e579ffd2cb59a10ca6e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cfe55a74bf5bf37209bafe91eccd2ce08f2a6b7fad4e579ffd2cb59a10ca6e7/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0e9c6e76c284c648e69d3a47ad0e3768220b5f1d65d1b2bab4650e306226393/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0e9c6e76c284c648e69d3a47ad0e3768220b5f1d65d1b2bab4650e306226393/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca1d6c132313e6dac8557a462d53ded2d50902cea7ef508f1d3a66bff60f6a1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca1d6c132313e6dac8557a462d53ded2d50902cea7ef508f1d3a66bff60f6a1a/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cafbd56f7d5a6e255efd7cad05bc4ce91951415b5d3a6a5e75dc8aead9accb9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cafbd56f7d5a6e255efd7cad05bc4ce91951415b5d3a6a5e75dc8aead9accb9b/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9f1e09d9f6517a4e994f7611016099cff16d476874e0f8d184520fd2bd9acc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9f1e09d9f6517a4e994f7611016099cff16d476874e0f8d184520fd2bd9acc6/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb7d0ea37f82d56890b42835a01bb482c4d38b3aa3eb8ffa0763c29e2aee3c01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb7d0ea37f82d56890b42835a01bb482c4d38b3aa3eb8ffa0763c29e2aee3c01/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02acb0f7-5c31-4397-ae32-fe076ef673e0/volumes/kubernetes.io~projected/kube-api-access-69cbg:{mountpoint:/var/lib/kubelet/pods/02acb0f7-5c31-4397-ae32-fe076ef673e0/volumes/kubernetes.io~projected/kube-api-access-69cbg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc8ab91-c338-4320-b006-574a196d2333/volumes/kubernetes.io~projected/kube-api-access-kz7s2:{mountpoint:/var/lib/kubelet/pods/0dc8ab91-c338-4320-b006-574a196d2333/volumes/kubernetes.io~projected/kube-api-access-kz7s2 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~projected/kube-api-access-8xqt4:{mountpoint:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~projected/kube-api-access-8xqt4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~projected/kube-api-access-wkzpm:{mountpoint:/var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~projected/kube-api-access-wkzpm major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~projected/kube-api-access-v76nk:{mountpoint:/var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~projected/kube-api-access-v76nk major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~secret/signing-key major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24bb5666-dc8c-400a-b112-508ea1a660de/volumes/kubernetes.io~projected/kube-api-access-6lr7b:{mountpoint:/var/lib/kubelet/pods/24bb5666-dc8c-400a-b112-508ea1a660de/volumes/kubernetes.io~projected/kube-api-access-6lr7b major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/volumes/kubernetes.io~projected/kube-api-access-nnjfq:{mountpoint:/var/lib/kubelet/pods/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/volumes/kubernetes.io~projected/kube-api-access-nnjfq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/kube-api-access-wbznk:{mountpoint:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/kube-api-access-wbznk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c533f37-a794-4b53-8779-86fbbb312bf6/volumes/kubernetes.io~projected/kube-api-access-t8stx:{mountpoint:/var/lib/kubelet/pods/5c533f37-a794-4b53-8779-86fbbb312bf6/volumes/kubernetes.io~projected/kube-api-access-t8stx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38/volumes/kubernetes.io~projected/kube-api-access-nzztw:{mountpoint:/var/lib/kubelet/pods/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38/volumes/kubernetes.io~projected/kube-api-access-nzztw major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~projected/kube-api-access-dr8r8:{mountpoint:/var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~projected/kube-api-access-dr8r8 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~projected/kube-api-access-n8ckg:{mountpoint:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~projected/kube-api-access-n8ckg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67/volumes/kubernetes.io~projected/kube-api-access-8mjgn:{mountpoint:/var/lib/kubelet/pods/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67/volumes/kubernetes.io~projected/kube-api-access-8mjgn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9251c254-07fe-47f7-9ca3-a0e00bec385c/volumes/kubernetes.io~projected/kube-api-access-mdsn7:{mountpoint:/var/lib/kubelet/pods/9251c254-07fe-47f7-9ca3-a0e00bec385c/volumes/kubernetes.io~projected/kube-api-access-mdsn7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92a37659-0e94-43e8-adb2-94ec2dfe09a1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/92a37659-0e94-43e8-adb2-94ec2dfe09a1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~projected/kube-api-access-vlcg4:{mountpoint:/var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~projected/kube-api-access-vlcg4 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c78665-fec7-466f-a032-51b497e7028a/volumes/kubernetes.io~projected/kube-api-access-2g62x:{mountpoint:/var/lib/kubelet/pods/98c78665-fec7-466f-a032-51b497e7028a/volumes/kubernetes.io~projected/kube-api-access-2g62x major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b036611-6b37-40c6-9a5d-01e3f42d6e1e/volumes/kubernetes.io~projected/kube-api-access-jqh7q:{mountpoint:/var/lib/kubelet/pods/9b036611-6b37-40c6-9a5d-01e3f42d6e1e/volumes/kubernetes.io~projected/kube-api-access-jqh7q major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~projected/kube-api-access-7w9ct:{mountpoint:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~projected/kube-api-access-7w9ct major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/default-certificate major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/stats-auth major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/kube-api-access-26gjk:{mountpoint:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/kube-api-access-26gjk major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b589e26c-f56e-426a-9c25-396f65e0488f/volumes/kubernetes.io~projected/kube-api-access-8dhj6:{mountpoint:/var/lib/kubelet/pods/b589e26c-f56e-426a-9c25-396f65e0488f/volumes/kubernetes.io~projected/kube-api-access-8dhj6 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~projected/kube-api-access-f57r2:{mountpoint:/var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~projected/kube-api-access-f57r2 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~projected/kube-api-access-8wxpz:{mountpoint:/var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~projected/kube-api-access-8wxpz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce0f4cff-6fee-4e71-b5d1-381028a7f628/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ce0f4cff-6fee-4e71-b5d1-381028a7f628/volumes/kubernetes.io~secret/agent-certs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e15f10-7732-42c8-a07e-dddaf048f664/volumes/kubernetes.io~projected/kube-api-access-xn7cs:{mountpoint:/var/lib/kubelet/pods/d0e15f10-7732-42c8-a07e-dddaf048f664/volumes/kubernetes.io~projected/kube-api-access-xn7cs major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6509794-be77-4085-aeec-92819c3c9041/volumes/kubernetes.io~projected/kube-api-access-pf9w7:{mountpoint:/var/lib/kubelet/pods/d6509794-be77-4085-aeec-92819c3c9041/volumes/kubernetes.io~projected/kube-api-access-pf9w7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da7bf2d6-e040-4695-aeb3-f0bb15293c98/volumes/kubernetes.io~projected/kube-api-access-5qsgw:{mountpoint:/var/lib/kubelet/pods/da7bf2d6-e040-4695-aeb3-f0bb15293c98/volumes/kubernetes.io~projected/kube-api-access-5qsgw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dac1afb9-e804-406f-9064-8fe8e976e0c6/volumes/kubernetes.io~projected/kube-api-access-jl4cq:{mountpoint:/var/lib/kubelet/pods/dac1afb9-e804-406f-9064-8fe8e976e0c6/volumes/kubernetes.io~projected/kube-api-access-jl4cq major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0777e96-1263-40e3-ad73-55e2a8eaa539/volumes/kubernetes.io~projected/kube-api-access-vhb66:{mountpoint:/var/lib/kubelet/pods/e0777e96-1263-40e3-ad73-55e2a8eaa539/volumes/kubernetes.io~projected/kube-api-access-vhb66 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191/volumes/kubernetes.io~projected/kube-api-access-jz6wk:{mountpoint:/var/lib/kubelet/pods/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191/volumes/kubernetes.io~projected/kube-api-access-jz6wk major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7174f1b-4841-4952-8f04-331347493639/volumes/kubernetes.io~projected/kube-api-access-h9fld:{mountpoint:/var/lib/kubelet/pods/e7174f1b-4841-4952-8f04-331347493639/volumes/kubernetes.io~projected/kube-api-access-h9fld major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb903f58-35c6-449e-8350-3382d4494692/volumes/kubernetes.io~projected/kube-api-access-j5584:{mountpoint:/var/lib/kubelet/pods/fb903f58-35c6-449e-8350-3382d4494692/volumes/kubernetes.io~projected/kube-api-access-j5584 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc51b84b-d859-4ec3-97ea-ec66bb8a8756/volumes/kubernetes.io~projected/kube-api-access-2br88:{mountpoint:/var/lib/kubelet/pods/fc51b84b-d859-4ec3-97ea-ec66bb8a8756/volumes/kubernetes.io~projected/kube-api-access-2br88 major:0 minor:325 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/39e42bc946c934fe41aa4d239b6586522a4a761878f9c0190cb2f701911551b9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a4da138d93cf0ad11daadc996a13b4b43e5a6312337c9e7f047a59ad5b2482a1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9b6c0270c1ea15ea136de2987aad4f2dbf46a527208f5ab551432587e6e3ecf6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e4e7de1907669bf148cd7959ed5b13cf900ab22c074ad5b414eb69a97cfbad90/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a4e53253c24f65e675e4aa10c4340879c453877d01a8befe6193c7b185be9c18/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/15f977e2651d7733fc6c1e9965f79633ceb3376dd8630ae7d750b2605845956c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4d253d5dfee442d8bcfe7ecbe9053a95a7baa5c43d5e724ecae74112fff99093/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e5160c1c2aa176a2d22d0b33b30a02dab349573245afae2bd32d7e3890d234ae/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c48a7bd595b695ddb6e4b56b52d5df6963ef21130b20c6976f790e48ee46ff33/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/424434130a44ac6cd8578bd93e6c0e982eb24ecede860fe5cff1d2f23b4f689e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/214589cbded6f4e68eefa175dea100c07d2ceb380d6eb6b195a3e6444de90e7b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/6485fe711a2cdc29d6eccd62d85c163f0635ca9dd1646ecb6197df481d466b9c/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e7aca2f59db3e5ff182c4c7148410ae9ac200e744eca04813d817ed6be61b04c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1ed93cbda9c0bd73b370fe7c601e2c2c8de329336281a7836299a87db2a54233/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fdd1c37c0ec3b15b158a8f54d1127949c0ee98ca44b93f0c2dccf7201487991d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/67881ebde13f942b31bf953931ab6269f92eac0b244c4c7abe098e351981945b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2a3ddc93781da84e4919541a95adce2eeae91a0daadcfdeac04aeb41d9e6b4a7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0e45f7890ae496eeba965ca87a8bd8a526ec59ff7445f5ad04fcc4b8414e2a33/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/833163859dc64763e881022c7d2f3fe29fc7b0235bd8ec023a17c25d7a907fbd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c58b86927f830515f3952a77bae928e51205a48eec6ca92c94a78bb590ba1544/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5a3ae0fa6683fbeb8befc29899fd69a59e3b33633ccc8bd177e057e5539e7eaa/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3ecfe7d6afa9eabfc06018b0c8eddfdf5065aee399bcebdb62726466c7e16c46/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/593b03420bb862484c5103c012a41dc81c2098d41367e67c86f5a77865269d1e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/5ebfde2d11b5a6c1bb85521da43fed47487a5baadb9694fb52bf6e12613882f3/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/927a157a333007d379ced5d2add666b665f85316486733791e600c40a94db058/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/2f4d2207252bff5bd521ff3fc64b4ac0433874bd91392e014cb769835bc696c3/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/590985efb1ef84999710e048323f543c0b18b0f7a3d37a229310f77642afe7d5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f30722f8f4af70ce83900751415895406c683ec34d39e67f9c8fd71e9dfe3532/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/adfb2180638e5bce3926281a28cbd3a1cf7504c90144a6b365d3a5a38e8b3e57/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/12a2202cded57a2a06473f7f3ff900e4cf0b442e3b7e80ce29c8a8588d158d74/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/a12dc09642ff40ab6b72d57ea865ca3424d7979c20e44ba232758d1990e46c16/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/156c9d55dab33740c2506f4becb2f17eab4a9be106f68411ccba1a817f35ba4a/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/b7c52e1b835dbcbbb3f7588e8e78c4dc3a7a507b9d38b040c081db9e5801a865/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3387c84e00a6ef6e2d60534d697b8f30288f0c5d024f432463caca97e43c382b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7b2b70a2cd4e60ae87fa04b41c00378d0a8b7ffda72a64beafc84e0b5562c12c/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/be8b2ce5065a1b4b4c894623f51acf55b3bfb59e6f3d0bc108fc030b51dc7c09/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/98273591f564803c40becfdf9821fb9ec9770f6d600eb46e3f8e27b3a9ca864f/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/5aeab4708bb161c622111fb8bc88b35686693c5e5356c1f39031a7ff45fb2267/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/faa4fca59b48466499d2f998d71cf0162be988dcc9a12270d91d48164b96956d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/f470536f9558fa1fe62a10f1197d80af2d64ebb7ea48c514ebd652c879c66dfe/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/a5440a3921a58f7e0c45c06c7933f180593b8401aec7b811d713c976a2b347ad/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f12887e4425fabe17737b90ba50230aa44931f34cd33ce29dcf5bd2aa3cf93cb/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/beed3380aeabd391990b0f693cd4912f9eb66f5f6e57a7af31771594976c795b/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/2cea877771d92e46f6c3c24ae44b2809d7829dd59dddd9d30016235ef2c7d853/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/2016b0d1b6bea84fe4a5d65fbe1ee348bcbd8b605d7b9a85a31303d2af545823/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a64fbae8c473a845bbfa61b10d0c952a15357140b349d166caa3ef853b0c837f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/cde87e1f361cb25b53334a3248258014095374c7c5ab40ec6bfa2933c08ab49b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/77fe14dd0489d4b3aa3cbf966086116463e3ffcfaf434a8c0ee0dacd0e479a99/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/b0b086e35ad4b048aa7bc5dc2504695da37c999e6b6a36c7447a30fbf54914fd/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/07f546cdd51c88d30558ff3a7720385b837f1f485b959ad2b82f1bb4d2eacdad/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/2f04684ca6f10d4d99bf1afeb40414c87825963c02cbf2b5dc740573aeee47dd/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/90c7a0837d84af6126d565902911487c4b6b8fc9eba915588149660250e5ab79/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/22c5f44660d2b172efa5d45d0283e2f598ee1ae50ee036e7f5bd9f4f08b9c416/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f2af9bb1f999ec6b231b287b4c343da77d8a7e328e61468ad727640fe74f17a5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/340dc3341132cf7c9d5357292f754ed561ffce8778af00a5b9974b5a31e0472e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c4ad20f6bcc3e44bbbcc767b6ac26603d526b106cce5f3e36a389e1a31dc5651/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c937e5ae9950874f5330e40e1300cd513aaee12efd80304858dc0189bf6285a9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9dc87d588f5f20ce7eb8a92e5f621d045be2a4edf8a396b6dec49a28e10cff4d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7a95c39225d432a10150ffebbbe6a78fe28e0c012b01d59b874718bcf79183fd/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f6ce37bea51326385a6de5f6c42cb2fa3deb3760321c2b3926564aa50211e10a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cc0bb3789a5ca2f87cfa5d12a89b346a4a3000a1efb82a975f03318024514a24/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fdb1b805e6183bf41480b562e9dfac6d7ef9dff1aea34d56216b5157e8284db1/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:39:36.121035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.120491 5036 manager.go:217] Machine: {Timestamp:2026-10-30 21:39:36.120012949 +0000 UTC m=+0.069967885 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119460 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f7fb92e2bc44abd82e05f5e33d17 SystemUUID:ec25f7fb-92e2-bc44-abd8-2e05f5e33d17 BootID:9f2d6868-facc-4992-bd22-a29288388f8d Filesystems:[{Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b54bdbd91d9936dc58fa891b5d7a8933a5b55148786014232ec2bd50eb3a5c37/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d2bf3d5e9fe511af5201608bf19854356ec59c288a9c5e2844a8fc484420c9b/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7174f1b-4841-4952-8f04-331347493639/volumes/kubernetes.io~projected/kube-api-access-h9fld DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/kube-api-access-26gjk DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38/volumes/kubernetes.io~projected/kube-api-access-nzztw DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc8ab91-c338-4320-b006-574a196d2333/volumes/kubernetes.io~projected/kube-api-access-kz7s2 DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6509794-be77-4085-aeec-92819c3c9041/volumes/kubernetes.io~projected/kube-api-access-pf9w7 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~projected/kube-api-access-8xqt4 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e15f10-7732-42c8-a07e-dddaf048f664/volumes/kubernetes.io~projected/kube-api-access-xn7cs DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86a1236b40644b1d6c9d13088f9aa84f4968e245ddce7a4600a08f887137037f/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce0f4cff-6fee-4e71-b5d1-381028a7f628/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67/volumes/kubernetes.io~projected/kube-api-access-8mjgn DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~projected/kube-api-access-7w9ct DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~projected/kube-api-access-8wxpz DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d2b8c98e15feb38ff78ecb5081d9e8766635477bf9bba1a1029cb49ffddbf38/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ed4c9aab8f358220b939d1e4376984f5a1d8792798ba6ffe17c0ffb5cc57211/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/volumes/kubernetes.io~projected/kube-api-access-nnjfq DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9829f19b-65f8-4435-a942-0c997ee268c3/volumes/kubernetes.io~projected/kube-api-access-vlcg4 DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0e9c6e76c284c648e69d3a47ad0e3768220b5f1d65d1b2bab4650e306226393/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~projected/kube-api-access-f57r2 DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~projected/kube-api-access-wkzpm DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c94c2d701ae91309d9d95657d1827c76e43fc8ec811c5609af0a6cd50006490/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9251c254-07fe-47f7-9ca3-a0e00bec385c/volumes/kubernetes.io~projected/kube-api-access-mdsn7 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24c75104031c053f16a8d0947315e3adcf9819e79a60382f8509e2419e03bc4f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fc51b84b-d859-4ec3-97ea-ec66bb8a8756/volumes/kubernetes.io~projected/kube-api-access-2br88 DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cf2f202d5d98b4ecfa128f5563c87f404360db025e2eedc22950d67451997bc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9f1e09d9f6517a4e994f7611016099cff16d476874e0f8d184520fd2bd9acc6/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95b6cae013a6053f6dacfa21e5f655793163a346447723089c2e80aab8b55d28/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0777e96-1263-40e3-ad73-55e2a8eaa539/volumes/kubernetes.io~projected/kube-api-access-vhb66 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb7d0ea37f82d56890b42835a01bb482c4d38b3aa3eb8ffa0763c29e2aee3c01/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98c78665-fec7-466f-a032-51b497e7028a/volumes/kubernetes.io~projected/kube-api-access-2g62x DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f04d9a58aca4ceb4400aafd130959525b830421473afcf8f93408aaadd1f9db/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b036611-6b37-40c6-9a5d-01e3f42d6e1e/volumes/kubernetes.io~projected/kube-api-access-jqh7q DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~projected/kube-api-access-n8ckg DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da7bf2d6-e040-4695-aeb3-f0bb15293c98/volumes/kubernetes.io~projected/kube-api-access-5qsgw DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~projected/kube-api-access-dr8r8 DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b218dc792c0e46ec7706395944584f8a1eb909c4fab93f4b54925a5cc44ac58/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cafbd56f7d5a6e255efd7cad05bc4ce91951415b5d3a6a5e75dc8aead9accb9b/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/74027051-12b8-4ebf-8e06-a9783a3c232c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/194d4b78-7c26-401b-858b-07730092cc53/volumes/kubernetes.io~projected/kube-api-access-v76nk DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cfe55a74bf5bf37209bafe91eccd2ce08f2a6b7fad4e579ffd2cb59a10ca6e7/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92a37659-0e94-43e8-adb2-94ec2dfe09a1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02acb0f7-5c31-4397-ae32-fe076ef673e0/volumes/kubernetes.io~projected/kube-api-access-69cbg DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c533f37-a794-4b53-8779-86fbbb312bf6/volumes/kubernetes.io~projected/kube-api-access-t8stx DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aea109c7dd9126f5085bdcd36c4ee703ca293f528079fa560e94e11d18e24407/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191/volumes/kubernetes.io~projected/kube-api-access-jz6wk DeviceMajor:0 DeviceMinor:324 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dac1afb9-e804-406f-9064-8fe8e976e0c6/volumes/kubernetes.io~projected/kube-api-access-jl4cq DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/841c9524b5670e8f274dd017f664f6a17b51038ffd415d6213ca91881612dd6a/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6b673e62-070f-4a1d-969c-1c4958789ba9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b589e26c-f56e-426a-9c25-396f65e0488f/volumes/kubernetes.io~projected/kube-api-access-8dhj6 DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50aaf34c67e95f79f5726bfd4069524e0ba1600f714367c337d5068c8d48de95/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/119335ed-3944-46c6-aacd-e63aff054eb8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc38fec3-2436-4e9e-aea8-4bf2fdab438e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f192bec7180b55fe501423f0d164a8e462cbff4d5272253603d85d4575285ed/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb903f58-35c6-449e-8350-3382d4494692/volumes/kubernetes.io~projected/kube-api-access-j5584 DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca1d6c132313e6dac8557a462d53ded2d50902cea7ef508f1d3a66bff60f6a1a/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c93cf8f00fe2636447610fb7d2774c8dfe470f6204dc687de17724d947636ef/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes/kubernetes.io~projected/kube-api-access-wbznk DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75d9b915b184808cef645b2ae64a91430b2c69b3b32587c9cbfafb30471ae79e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/392e825ea34aef509e92d5ee17f37a5e8174d96e820d7563ef83e9dd1e6a0712/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6c881db-f097-4bf2-af04-3aa394bdfc44/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24bb5666-dc8c-400a-b112-508ea1a660de/volumes/kubernetes.io~projected/kube-api-access-6lr7b DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24c75104031c053 MacAddress:5e:fc:f3:bd:3e:ec Speed:10000 Mtu:8901} {Name:3f04d9a58aca4ce MacAddress:4e:e9:47:78:b9:6b Speed:10000 Mtu:8901} {Name:4d2b8c98e15feb3 MacAddress:26:b3:19:27:b0:da Speed:10000 Mtu:8901} {Name:5d2bf3d5e9fe511 MacAddress:6e:cf:23:7b:04:09 Speed:10000 Mtu:8901} {Name:6b218dc792c0e46 MacAddress:c6:7e:d5:cf:cd:35 Speed:10000 Mtu:8901} {Name:841c9524b5670e8 MacAddress:ba:49:14:4c:70:c2 Speed:10000 Mtu:8901} {Name:86a1236b40644b1 MacAddress:76:fe:89:88:26:b0 Speed:10000 Mtu:8901} {Name:8c94c2d701ae913 MacAddress:b2:7c:73:0b:d2:0e Speed:10000 Mtu:8901} {Name:8cfe55a74bf5bf3 MacAddress:d6:b3:75:16:f0:05 Speed:10000 Mtu:8901} {Name:95b6cae013a6053 MacAddress:32:55:34:22:47:66 Speed:10000 Mtu:8901} {Name:9ed4c9aab8f3582 MacAddress:da:a5:43:b2:ad:dd Speed:10000 Mtu:8901} {Name:b0e9c6e76c284c6 MacAddress:66:d5:e8:9d:3a:5f Speed:10000 Mtu:8901} {Name:b54bdbd91d9936d MacAddress:46:7a:d6:f2:b3:47 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:ee:a3:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:6c:f6:a0:86:52 Speed:0 Mtu:8901} {Name:ca1d6c132313e6d MacAddress:be:99:99:8b:da:81 Speed:10000 Mtu:8901} {Name:cafbd56f7d5a6e2 MacAddress:7e:82:5f:2e:ff:e7 Speed:10000 Mtu:8901} {Name:eb7d0ea37f82d56 MacAddress:1a:5d:26:e5:62:17 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:ee:a3:11 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7e:ff:25:06:61:ad Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:59:d3:cf:9b:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:39:36.121035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.120851 5036 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:39:36.121035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.120890 5036 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:39:36.121290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121160 5036 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:39:36.121334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121176 5036 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-193.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:39:36.121383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121336 5036 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:39:36.121383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121345 5036 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:39:36.121383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121357 5036 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:39:36.121383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121376 5036 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:39:36.121483 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121395 5036 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:39:36.121573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121563 5036 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:39:36.121681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121672 5036 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:39:36.121712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121708 5036 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:39:36.121739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121718 5036 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:39:36.121739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121727 5036 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:39:36.121739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121736 5036 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:39:36.121824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.121745 5036 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:39:36.123117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.122810 5036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:39:36.123117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.122829 5036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:39:36.123747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.123533 5036 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:39:36.127433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127413 5036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:39:36.127433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127431 5036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:39:36.127547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127535 5036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:39:36.127735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127724 5036 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:39:36.127775 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127754 5036 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:39:36.127865 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127856 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:39:36.127895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127871 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:39:36.127895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127877 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:39:36.127895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127883 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:39:36.127895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127892 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127902 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127910 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127915 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127922 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127928 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127936 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127946 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127970 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:39:36.128035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.127978 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:39:36.131409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.131396 5036 server.go:1263] "Started kubelet" Oct 30 21:39:36.131530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.131477 5036 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:39:36.131588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.131520 5036 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:39:36.132004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.131632 5036 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:39:36.132004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.131977 5036 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:39:36.132080 ip-10-0-133-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:39:36.134169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.134145 5036 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:39:36.137575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.137124 5036 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:36.139100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.138342 5036 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:36.139100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.138628 5036 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:39:36.139100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.138674 5036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:39:36.139100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.138947 5036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:33:56 +0000 UTC" deadline="2028-08-08 07:52:15.275612682 +0000 UTC" Oct 30 21:39:36.139100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.138990 5036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15538h12m39.136629013s" Oct 30 21:39:36.139418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.139149 5036 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:39:36.139418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.139165 5036 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:39:36.139749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.139703 5036 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141173 5036 factory.go:55] Registering systemd factory Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141194 5036 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141491 5036 factory.go:153] Registering CRI-O factory Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141503 5036 factory.go:223] Registration of the crio container factory successfully Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141584 5036 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141639 5036 factory.go:103] Registering Raw factory Oct 30 21:39:36.141724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141659 5036 manager.go:1199] Started watching for new ooms in manager Oct 30 21:39:36.142084 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.141983 5036 manager.go:319] Starting recovery of all containers Oct 30 21:39:36.143869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143828 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143873 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dac1afb9-e804-406f-9064-8fe8e976e0c6" volumeName="kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143888 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7174f1b-4841-4952-8f04-331347493639" volumeName="kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143902 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d" volumeName="kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143917 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c533f37-a794-4b53-8779-86fbbb312bf6" volumeName="kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143930 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9829f19b-65f8-4435-a942-0c997ee268c3" volumeName="kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143945 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:39:36.143959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143957 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0777e96-1263-40e3-ad73-55e2a8eaa539" volumeName="kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143973 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d" volumeName="kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.143987 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74027051-12b8-4ebf-8e06-a9783a3c232c" volumeName="kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144001 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b036611-6b37-40c6-9a5d-01e3f42d6e1e" volumeName="kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144015 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144029 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc38fec3-2436-4e9e-aea8-4bf2fdab438e" volumeName="kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144044 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d" volumeName="kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144072 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="675ec9b6-88b1-4fe3-a589-a0bcd26bfb38" volumeName="kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144086 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144101 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67" volumeName="kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144127 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c" volumeName="kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144167 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0777e96-1263-40e3-ad73-55e2a8eaa539" volumeName="kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144181 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="194d4b78-7c26-401b-858b-07730092cc53" volumeName="kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144194 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c78665-fec7-466f-a032-51b497e7028a" volumeName="kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144207 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144223 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dac1afb9-e804-406f-9064-8fe8e976e0c6" volumeName="kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144237 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0777e96-1263-40e3-ad73-55e2a8eaa539" volumeName="kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144250 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119335ed-3944-46c6-aacd-e63aff054eb8" volumeName="kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144280 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4763a8d1-c6f0-49eb-81e3-a6919c5f4a85" volumeName="kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144295 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67" volumeName="kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn" seLinuxMountContext="" Oct 30 21:39:36.144303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144309 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c881db-f097-4bf2-af04-3aa394bdfc44" volumeName="kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144330 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc38fec3-2436-4e9e-aea8-4bf2fdab438e" volumeName="kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144343 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" volumeName="kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144371 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144384 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144417 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b589e26c-f56e-426a-9c25-396f65e0488f" volumeName="kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144431 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c881db-f097-4bf2-af04-3aa394bdfc44" volumeName="kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144446 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce0f4cff-6fee-4e71-b5d1-381028a7f628" volumeName="kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144459 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" volumeName="kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144472 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d" volumeName="kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144487 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="194d4b78-7c26-401b-858b-07730092cc53" volumeName="kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144500 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="194d4b78-7c26-401b-858b-07730092cc53" volumeName="kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144513 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144532 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144546 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9251c254-07fe-47f7-9ca3-a0e00bec385c" volumeName="kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144561 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc38fec3-2436-4e9e-aea8-4bf2fdab438e" volumeName="kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144574 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce0f4cff-6fee-4e71-b5d1-381028a7f628" volumeName="kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144587 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc8ab91-c338-4320-b006-574a196d2333" volumeName="kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144600 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4763a8d1-c6f0-49eb-81e3-a6919c5f4a85" volumeName="kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144617 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144632 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74027051-12b8-4ebf-8e06-a9783a3c232c" volumeName="kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144646 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c" volumeName="kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144659 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e33c44f7-69f3-4f6e-bbfd-760ba6440f2c" volumeName="kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144673 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" volumeName="kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144686 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9829f19b-65f8-4435-a942-0c997ee268c3" volumeName="kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144702 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119335ed-3944-46c6-aacd-e63aff054eb8" volumeName="kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144716 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c533f37-a794-4b53-8779-86fbbb312bf6" volumeName="kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144730 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9251c254-07fe-47f7-9ca3-a0e00bec385c" volumeName="kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144744 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144768 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6509794-be77-4085-aeec-92819c3c9041" volumeName="kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144782 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4763a8d1-c6f0-49eb-81e3-a6919c5f4a85" volumeName="kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144797 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74027051-12b8-4ebf-8e06-a9783a3c232c" volumeName="kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144811 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da7bf2d6-e040-4695-aeb3-f0bb15293c98" volumeName="kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144823 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" volumeName="kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144848 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc51b84b-d859-4ec3-97ea-ec66bb8a8756" volumeName="kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144862 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144875 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144900 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9251c254-07fe-47f7-9ca3-a0e00bec385c" volumeName="kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144914 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a37659-0e94-43e8-adb2-94ec2dfe09a1" volumeName="kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144927 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9829f19b-65f8-4435-a942-0c997ee268c3" volumeName="kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144938 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e15f10-7732-42c8-a07e-dddaf048f664" volumeName="kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144951 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6509794-be77-4085-aeec-92819c3c9041" volumeName="kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.144981 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc51b84b-d859-4ec3-97ea-ec66bb8a8756" volumeName="kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145018 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02acb0f7-5c31-4397-ae32-fe076ef673e0" volumeName="kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145030 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24bb5666-dc8c-400a-b112-508ea1a660de" volumeName="kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145044 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145066 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b673e62-070f-4a1d-969c-1c4958789ba9" volumeName="kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145079 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9251c254-07fe-47f7-9ca3-a0e00bec385c" volumeName="kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145091 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c" volumeName="kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145113 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145134 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9829f19b-65f8-4435-a942-0c997ee268c3" volumeName="kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145146 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145167 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c881db-f097-4bf2-af04-3aa394bdfc44" volumeName="kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145180 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb903f58-35c6-449e-8350-3382d4494692" volumeName="kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145211 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb903f58-35c6-449e-8350-3382d4494692" volumeName="kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584" seLinuxMountContext="" Oct 30 21:39:36.145187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145228 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145241 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adcfff7f-94e9-4cb3-b800-acdc636274ab" volumeName="kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token" seLinuxMountContext="" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145254 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d" volumeName="kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides" seLinuxMountContext="" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145294 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f7e4a-14f0-4e90-819a-118a0ea2c058" volumeName="kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk" seLinuxMountContext="" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145306 5036 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145313 5036 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:39:36.147649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.145580 5036 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:36.149578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.149561 5036 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:39:36.157963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.157939 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:36.162783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.162759 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:36.177663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.177632 5036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:39:36.179490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.179469 5036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:39:36.179642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.179496 5036 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:39:36.179642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.179517 5036 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:39:36.179642 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:36.179557 5036 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:39:36.182870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.182850 5036 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:36.229590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.229551 5036 generic.go:356] "Generic (PLEG): container finished" podID="b9317762339865a73d28250c9a6a406c" containerID="7c4763f32145e84702a8bdea1def3e35b4b741cb33b8a0d0d424e3976ea41918" exitCode=0 Oct 30 21:39:36.236211 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.236189 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:36.236384 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.236218 5036 generic.go:356] "Generic (PLEG): container finished" podID="9829f19b-65f8-4435-a942-0c997ee268c3" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" exitCode=255 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242500 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="58adfc689c6f977598445dc2d342255582e1ba8bc8644176ba8342f5b5489c9d" exitCode=0 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242523 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="a40867d735d32c11aeb909cdd89fab8b9c8babf21ff09059ef47a118b8198089" exitCode=0 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242532 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="a45eae0eeb1f0fee53e490e0c8afe6ee13c499721a635bbe1ab3164695316e25" exitCode=0 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242540 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="780b7394a5c9b950b8d8f4c9422cbedf8de957693e63353049be8d9777580aa1" exitCode=0 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242546 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="c73bb4a005f56ea93ed118db2d888c97b383c4ec74ffaebbaf1a8be2595437c5" exitCode=0 Oct 30 21:39:36.242558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.242552 5036 generic.go:356] "Generic (PLEG): container finished" podID="9251c254-07fe-47f7-9ca3-a0e00bec385c" containerID="cdfb3cb28a8380cfe640c097ab9ac2a608c579de40341839037ccb5b324b8de0" exitCode=0 Oct 30 21:39:36.245163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.245145 5036 manager.go:324] Recovery completed Oct 30 21:39:36.270094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270070 5036 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:39:36.270094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270085 5036 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:39:36.270241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270103 5036 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:39:36.270241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270220 5036 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:39:36.270724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270231 5036 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:39:36.270756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270729 5036 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:39:36.270756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270736 5036 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:39:36.270756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270742 5036 policy_none.go:50] "Start" Oct 30 21:39:36.270756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270751 5036 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:39:36.270864 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270765 5036 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:39:36.270924 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270914 5036 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:39:36.270953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270925 5036 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:39:36.270953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.270932 5036 policy_none.go:44] "Start" Oct 30 21:39:36.275609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.275581 5036 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:39:36.278303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.278286 5036 manager.go:341] "Starting Device Plugin manager" Oct 30 21:39:36.278402 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:36.278316 5036 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:39:36.278402 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.278328 5036 server.go:87] "Starting device plugin registration server" Oct 30 21:39:36.278635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.278621 5036 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:39:36.278702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.278635 5036 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:39:36.278748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.278724 5036 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:39:36.279917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.279877 5036 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:39:36.280089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.279927 5036 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:39:36.280089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.279934 5036 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:39:36.280089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.279910 5036 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal"] Oct 30 21:39:36.280089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.279978 5036 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:39:36.279915469Z","UUID":"496262a1-c454-424d-afd1-81ad1c79ce7e","Handler":null,"Name":"","Endpoint":""} Oct 30 21:39:36.280306 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:36.280259 5036 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:39:36.280679 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280647 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"05953060daddf043bb6b311d0281468b","Type":"ContainerStarted","Data":"9645f546bd6c79a4ef4207d53791ade632e0b93ab43ed639a4b935c3e01780e3"} Oct 30 21:39:36.280815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280691 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"05953060daddf043bb6b311d0281468b","Type":"ContainerStarted","Data":"a612af7bfd946dab38d80c63304bcf27026ea1f2c8ee11174be4a891f7932ed5"} Oct 30 21:39:36.280815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerStarted","Data":"b589c2d51fae821a7abcf4e12fb3c983e5bbdd6b4fadb2c77cb4bee07b36a164"} Oct 30 21:39:36.280917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280824 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerDied","Data":"7c4763f32145e84702a8bdea1def3e35b4b741cb33b8a0d0d424e3976ea41918"} Oct 30 21:39:36.280917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280846 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.281061 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.281045 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.281162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.280848 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" event={"ID":"b9317762339865a73d28250c9a6a406c","Type":"ContainerStarted","Data":"fd6511b78c1ea5289f57ade2d14b3a6ab7f9ac894d46ab407602c911f7a53803"} Oct 30 21:39:36.281480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.281465 5036 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:39:36.281556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.281501 5036 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:39:36.295176 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.295157 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:39:36.295299 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:36.295207 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.296028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.296013 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:39:36.296105 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:36.296056 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.350378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.350341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.350378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.350377 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.350553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.350397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.379454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.379422 5036 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:36.381617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.381597 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:36.381710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.381629 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:36.381710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.381648 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:36.381710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.381692 5036 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.393130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.393112 5036 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.393196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.393169 5036 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450773 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450657 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450773 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450773 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450735 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05953060daddf043bb6b311d0281468b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"05953060daddf043bb6b311d0281468b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450773 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450743 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:36.450954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:36.450743 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9317762339865a73d28250c9a6a406c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" (UID: \"b9317762339865a73d28250c9a6a406c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:37.122195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.122162 5036 apiserver.go:52] "Watching apiserver" Oct 30 21:39:37.190851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.190823 5036 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:39:37.191719 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.191692 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48","openshift-insights/insights-operator-7c4986d449-wh95c","openshift-network-diagnostics/network-check-target-dr5t5","kube-system/kas-connection-checker-685895c7c9-r7rws","openshift-insights/insights-runtime-extractor-jqzzl","kube-system/kas-connection-checker-685895c7c9-9jmb6","openshift-console-operator/console-operator-7cd56b8dbc-dljwb","openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht","openshift-ingress/router-default-7df45874d9-82hkt","openshift-network-operator/iptables-alerter-9qwxg","openshift-multus/multus-hk8sl","openshift-image-registry/image-registry-875fcbc87-fc57r","openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx","openshift-network-console/networking-console-plugin-86c478bb88-92mlp","openshift-insights/periodic-gathering-4lbx6-g9gk4","openshift-image-registry/image-registry-76d5dcc446-9v622","kube-system/kas-connection-checker-685895c7c9-kwwwz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-xcg97","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5","openshift-service-ca/service-ca-5fffc97bdf-tkt5h","openshift-multus/multus-additional-cni-plugins-7pq9d","openshift-ovn-kubernetes/ovnkube-node-zfdzh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw","kube-system/konnectivity-agent-xfnx9","openshift-dns/dns-default-hm8lm","openshift-dns/node-resolver-h7xwb","openshift-ingress-canary/ingress-canary-fxkw8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9","kube-system/kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n","openshift-image-registry/node-ca-pgsq6","openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p","kube-system/global-pull-secret-syncer-nd669","openshift-cluster-node-tuning-operator/tuned-slpv5"] Oct 30 21:39:37.195631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.195590 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.196935 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.196915 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.197123 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.197103 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.197231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.197192 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.198118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.198096 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.199815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.199795 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.200299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.200281 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.200647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.200632 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.200776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.200758 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.201151 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.201131 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:39:37.201248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.201228 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:39:37.201466 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.201451 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xzsv2\"" Oct 30 21:39:37.201803 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.201788 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:39:37.203589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.203564 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:39:37.203714 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.203699 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:39:37.203767 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.203729 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.205467 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205448 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.205533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205468 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.205533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205515 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.205641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205623 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:39:37.205838 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205819 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.205899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.205883 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:39:37.206132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206115 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:39:37.206205 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206165 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.206287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206224 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.206287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206280 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206324 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206339 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206342 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206354 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206366 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2mzw9\"" Oct 30 21:39:37.206379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206374 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:39:37.206622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206406 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.206622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206406 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.206622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.206417 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.207081 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.207066 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.207551 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.207526 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.211126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.211108 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.216372 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.216357 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:39:37.216669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.216656 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:39:37.217711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.217698 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rvhrj\"" Oct 30 21:39:37.217751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.217705 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:39:37.217751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.217720 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.217751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.217701 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:39:37.218332 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.218318 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l6nmx\"" Oct 30 21:39:37.218470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.218456 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:39:37.218690 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.218679 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.219059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219046 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tqvfj\"" Oct 30 21:39:37.219110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219077 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wpbfh\"" Oct 30 21:39:37.219110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219084 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gx2bs\"" Oct 30 21:39:37.219381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219365 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:39:37.219454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219406 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8td58\"" Oct 30 21:39:37.219510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219455 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d8rvn\"" Oct 30 21:39:37.219510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219476 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:39:37.219776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.219761 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4qw75\"" Oct 30 21:39:37.225369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.225349 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:39:37.225449 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.225385 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:39:37.225600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.225586 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:39:37.227640 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.227624 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:39:37.227722 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.227702 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9m67w\"" Oct 30 21:39:37.230177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.230160 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:39:37.230554 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.230539 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.230612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.230598 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:39:37.230653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.230607 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.230653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.230624 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.231153 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231136 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.231203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231158 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\"" Oct 30 21:39:37.231203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231179 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\"" Oct 30 21:39:37.231203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231181 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:39:37.231318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231197 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.231512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231469 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\"" Oct 30 21:39:37.231512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231491 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:39:37.231512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231503 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:39:37.231647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231534 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\"" Oct 30 21:39:37.231702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231684 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\"" Oct 30 21:39:37.231750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231690 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:39:37.231863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231848 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jntk6\"" Oct 30 21:39:37.231904 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231879 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-csxns\"" Oct 30 21:39:37.231958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.231947 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.232345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232332 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:39:37.232414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232382 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:39:37.232480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232467 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.232531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232491 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:39:37.232531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232501 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\"" Oct 30 21:39:37.232531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232499 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.232531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232525 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45g4h\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232526 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232540 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232547 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-glzqf\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232540 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232542 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232577 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232582 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232604 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232608 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232620 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232616 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:39:37.232677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232665 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\"" Oct 30 21:39:37.232984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232715 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\"" Oct 30 21:39:37.232984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232755 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xqfqq\"" Oct 30 21:39:37.232984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232783 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\"" Oct 30 21:39:37.232984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232962 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:39:37.233121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.232995 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\"" Oct 30 21:39:37.233121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.233012 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:39:37.233475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.233433 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.233475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.233452 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:39:37.233475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.233459 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:39:37.234813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.234801 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.234990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.234969 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.234990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.234971 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:39:37.235094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235017 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.235094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235027 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:39:37.235293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235279 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:39:37.235293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235287 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235296 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235312 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235313 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235329 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235340 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235370 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:39:37.235409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235390 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:39:37.235730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.235632 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:39:37.236431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.236419 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:37.241922 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.241903 5036 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:39:37.247173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.247142 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:37.254758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.254829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254761 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.254829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254778 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cbg\" (UniqueName: \"kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.254829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5584\" (UniqueName: \"kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.254953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.255000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254973 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzpm\" (UniqueName: \"kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:37.255029 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.254998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.255029 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255021 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.255098 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255043 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.255150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.255198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255148 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-config-volume\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.255198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255172 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-cni-binary-copy\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.255198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255148 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.255354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255215 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-tmp\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.255354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255220 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.255354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255253 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9ct\" (UniqueName: \"kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.255493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255356 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.255493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255393 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.255493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255436 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.255493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fld\" (UniqueName: \"kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld\") pod \"migrator-66df5d496b-5h44n\" (UID: \"e7174f1b-4841-4952-8f04-331347493639\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255498 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255498 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255523 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-script-lib\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ckg\" (UniqueName: \"kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57r2\" (UniqueName: \"kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255609 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255668 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255701 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/194d4b78-7c26-401b-858b-07730092cc53-signing-key\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.255731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255745 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255783 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255820 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255939 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjfq\" (UniqueName: \"kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.255979 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256021 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9w7\" (UniqueName: \"kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-data-volume\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256072 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256096 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256114 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjgn\" (UniqueName: \"kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256129 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256148 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256191 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:37.256210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256209 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6c881db-f097-4bf2-af04-3aa394bdfc44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256303 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256334 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256360 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovnkube-config\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256384 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256438 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/119335ed-3944-46c6-aacd-e63aff054eb8-cert\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256532 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256594 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256687 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256753 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqt4\" (UniqueName: \"kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256777 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256804 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g62x\" (UniqueName: \"kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x\") pod \"kas-connection-checker-685895c7c9-r7rws\" (UID: \"98c78665-fec7-466f-a032-51b497e7028a\") " pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256838 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256859 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.256996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.256968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257030 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257058 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-serving-cert\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257073 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-stats-auth\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257127 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlcg4\" (UniqueName: \"kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257185 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257310 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257346 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257449 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4cq\" (UniqueName: \"kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-snapshots\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257484 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257499 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9829f19b-65f8-4435-a942-0c997ee268c3-serving-cert\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxpz\" (UniqueName: \"kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257557 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257571 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.257638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257764 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7cs\" (UniqueName: \"kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs\") pod \"migrator-66df5d496b-rlsht\" (UID: \"d0e15f10-7732-42c8-a07e-dddaf048f664\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257803 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257824 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257862 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4cff-6fee-4e71-b5d1-381028a7f628-konnectivity-ca\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.257989 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhb66\" (UniqueName: \"kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258029 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258048 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258136 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258159 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-daemon-config\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258166 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.258286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258217 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-env-overrides\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258340 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c533f37-a794-4b53-8779-86fbbb312bf6-tmp-dir\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258349 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b673e62-070f-4a1d-969c-1c4958789ba9-service-ca-bundle\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258391 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b673e62-070f-4a1d-969c-1c4958789ba9-tmp\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258537 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258568 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258585 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8r8\" (UniqueName: \"kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258628 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258682 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258707 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258733 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258763 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/194d4b78-7c26-401b-858b-07730092cc53-signing-cabundle\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-config\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258790 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258793 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce0f4cff-6fee-4e71-b5d1-381028a7f628-agent-certs\") pod \"konnectivity-agent-xfnx9\" (UID: \"ce0f4cff-6fee-4e71-b5d1-381028a7f628\") " pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258820 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.258882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258905 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258933 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.258970 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259017 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259040 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92a37659-0e94-43e8-adb2-94ec2dfe09a1-original-pull-secret\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259060 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-default-certificate\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dac1afb9-e804-406f-9064-8fe8e976e0c6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259183 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259213 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259228 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259279 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259317 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lr7b\" (UniqueName: \"kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b\") pod \"network-check-source-6f78d59f99-hqp48\" (UID: \"24bb5666-dc8c-400a-b112-508ea1a660de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259358 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsgw\" (UniqueName: \"kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259482 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6509794-be77-4085-aeec-92819c3c9041-serviceca\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259538 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzztw\" (UniqueName: \"kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw\") pod \"volume-data-source-validator-58c9f5f46d-zn459\" (UID: \"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259566 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259590 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259616 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259641 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259649 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.259645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259714 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v76nk\" (UniqueName: \"kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259805 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259812 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259917 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259938 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8stx\" (UniqueName: \"kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.259998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh7q\" (UniqueName: \"kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q\") pod \"kas-connection-checker-685895c7c9-kwwwz\" (UID: \"9b036611-6b37-40c6-9a5d-01e3f42d6e1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260021 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260050 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260286 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260324 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260451 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260463 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260524 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c881db-f097-4bf2-af04-3aa394bdfc44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260530 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260585 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260613 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br88\" (UniqueName: \"kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260639 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsn7\" (UniqueName: \"kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260666 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260665 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260684 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9251c254-07fe-47f7-9ca3-a0e00bec385c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260694 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260711 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.260731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b673e62-070f-4a1d-969c-1c4958789ba9-serving-cert\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260899 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-iptables-alerter-script\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260883 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.260993 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261277 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261306 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261337 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261346 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-tmp-dir\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261378 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhj6\" (UniqueName: \"kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6\") pod \"kas-connection-checker-685895c7c9-9jmb6\" (UID: \"b589e26c-f56e-426a-9c25-396f65e0488f\") " pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261394 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-tuned\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261400 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261416 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9829f19b-65f8-4435-a942-0c997ee268c3-config\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261427 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261453 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.261840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261847 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.262539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.261923 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:39:37.262539 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.261969 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.us-east-2.compute.internal" Oct 30 21:39:37.271574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.271558 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7cs\" (UniqueName: \"kubernetes.io/projected/d0e15f10-7732-42c8-a07e-dddaf048f664-kube-api-access-xn7cs\") pod \"migrator-66df5d496b-rlsht\" (UID: \"d0e15f10-7732-42c8-a07e-dddaf048f664\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rlsht" Oct 30 21:39:37.272695 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.272674 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cbg\" (UniqueName: \"kubernetes.io/projected/02acb0f7-5c31-4397-ae32-fe076ef673e0-kube-api-access-69cbg\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.272773 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.272707 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjfq\" (UniqueName: \"kubernetes.io/projected/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-kube-api-access-nnjfq\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.272836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.272823 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5584\" (UniqueName: \"kubernetes.io/projected/fb903f58-35c6-449e-8350-3382d4494692-kube-api-access-j5584\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.273040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.273021 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxpz\" (UniqueName: \"kubernetes.io/projected/cc38fec3-2436-4e9e-aea8-4bf2fdab438e-kube-api-access-8wxpz\") pod \"service-ca-operator-544d749c8c-ltt4p\" (UID: \"cc38fec3-2436-4e9e-aea8-4bf2fdab438e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" Oct 30 21:39:37.273097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.273062 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqt4\" (UniqueName: \"kubernetes.io/projected/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-kube-api-access-8xqt4\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.275237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.275221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ckg\" (UniqueName: \"kubernetes.io/projected/74027051-12b8-4ebf-8e06-a9783a3c232c-kube-api-access-n8ckg\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.277136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.277117 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g62x\" (UniqueName: \"kubernetes.io/projected/98c78665-fec7-466f-a032-51b497e7028a-kube-api-access-2g62x\") pod \"kas-connection-checker-685895c7c9-r7rws\" (UID: \"98c78665-fec7-466f-a032-51b497e7028a\") " pod="kube-system/kas-connection-checker-685895c7c9-r7rws" Oct 30 21:39:37.277398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.277383 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8stx\" (UniqueName: \"kubernetes.io/projected/5c533f37-a794-4b53-8779-86fbbb312bf6-kube-api-access-t8stx\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.283225 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.283200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlcg4\" (UniqueName: \"kubernetes.io/projected/9829f19b-65f8-4435-a942-0c997ee268c3-kube-api-access-vlcg4\") pod \"console-operator-7cd56b8dbc-dljwb\" (UID: \"9829f19b-65f8-4435-a942-0c997ee268c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:37.283312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.283207 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lr7b\" (UniqueName: \"kubernetes.io/projected/24bb5666-dc8c-400a-b112-508ea1a660de-kube-api-access-6lr7b\") pod \"network-check-source-6f78d59f99-hqp48\" (UID: \"24bb5666-dc8c-400a-b112-508ea1a660de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hqp48" Oct 30 21:39:37.283956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.283937 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v76nk\" (UniqueName: \"kubernetes.io/projected/194d4b78-7c26-401b-858b-07730092cc53-kube-api-access-v76nk\") pod \"service-ca-5fffc97bdf-tkt5h\" (UID: \"194d4b78-7c26-401b-858b-07730092cc53\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tkt5h" Oct 30 21:39:37.284100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284078 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhb66\" (UniqueName: \"kubernetes.io/projected/e0777e96-1263-40e3-ad73-55e2a8eaa539-kube-api-access-vhb66\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.284472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284452 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhj6\" (UniqueName: \"kubernetes.io/projected/b589e26c-f56e-426a-9c25-396f65e0488f-kube-api-access-8dhj6\") pod \"kas-connection-checker-685895c7c9-9jmb6\" (UID: \"b589e26c-f56e-426a-9c25-396f65e0488f\") " pod="kube-system/kas-connection-checker-685895c7c9-9jmb6" Oct 30 21:39:37.284552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsgw\" (UniqueName: \"kubernetes.io/projected/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kube-api-access-5qsgw\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.284642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284575 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.284642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57r2\" (UniqueName: \"kubernetes.io/projected/c6c881db-f097-4bf2-af04-3aa394bdfc44-kube-api-access-f57r2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw\" (UID: \"c6c881db-f097-4bf2-af04-3aa394bdfc44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" Oct 30 21:39:37.284778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.284761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.285240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.285221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7s2\" (UniqueName: \"kubernetes.io/projected/0dc8ab91-c338-4320-b006-574a196d2333-kube-api-access-kz7s2\") pod \"network-check-target-dr5t5\" (UID: \"0dc8ab91-c338-4320-b006-574a196d2333\") " pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:37.288597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.288574 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsn7\" (UniqueName: \"kubernetes.io/projected/9251c254-07fe-47f7-9ca3-a0e00bec385c-kube-api-access-mdsn7\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.288726 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.288574 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fld\" (UniqueName: \"kubernetes.io/projected/e7174f1b-4841-4952-8f04-331347493639-kube-api-access-h9fld\") pod \"migrator-66df5d496b-5h44n\" (UID: \"e7174f1b-4841-4952-8f04-331347493639\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5h44n" Oct 30 21:39:37.289160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.289292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289144 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9w7\" (UniqueName: \"kubernetes.io/projected/d6509794-be77-4085-aeec-92819c3c9041-kube-api-access-pf9w7\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.289292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289228 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br88\" (UniqueName: \"kubernetes.io/projected/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-api-access-2br88\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.289437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289412 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjgn\" (UniqueName: \"kubernetes.io/projected/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-kube-api-access-8mjgn\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.289527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh7q\" (UniqueName: \"kubernetes.io/projected/9b036611-6b37-40c6-9a5d-01e3f42d6e1e-kube-api-access-jqh7q\") pod \"kas-connection-checker-685895c7c9-kwwwz\" (UID: \"9b036611-6b37-40c6-9a5d-01e3f42d6e1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kwwwz" Oct 30 21:39:37.289957 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289939 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzztw\" (UniqueName: \"kubernetes.io/projected/675ec9b6-88b1-4fe3-a589-a0bcd26bfb38-kube-api-access-nzztw\") pod \"volume-data-source-validator-58c9f5f46d-zn459\" (UID: \"675ec9b6-88b1-4fe3-a589-a0bcd26bfb38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zn459" Oct 30 21:39:37.290036 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.289984 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4cq\" (UniqueName: \"kubernetes.io/projected/dac1afb9-e804-406f-9064-8fe8e976e0c6-kube-api-access-jl4cq\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.290322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.290303 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzpm\" (UniqueName: \"kubernetes.io/projected/119335ed-3944-46c6-aacd-e63aff054eb8-kube-api-access-wkzpm\") pod \"ingress-canary-fxkw8\" (UID: \"119335ed-3944-46c6-aacd-e63aff054eb8\") " pod="openshift-ingress-canary/ingress-canary-fxkw8" Oct 30 21:39:37.290380 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.290368 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.290496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.290477 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8r8\" (UniqueName: \"kubernetes.io/projected/6b673e62-070f-4a1d-969c-1c4958789ba9-kube-api-access-dr8r8\") pod \"insights-operator-7c4986d449-wh95c\" (UID: \"6b673e62-070f-4a1d-969c-1c4958789ba9\") " pod="openshift-insights/insights-operator-7c4986d449-wh95c" Oct 30 21:39:37.290834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.290818 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9ct\" (UniqueName: \"kubernetes.io/projected/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-kube-api-access-7w9ct\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.290875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.290861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"periodic-gathering-4lbx6-g9gk4\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:39:37.362346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362321 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362350 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362367 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362383 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362401 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-var-lib-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362432 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362454 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-crio-socket\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362479 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-kubelet\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-socket-dir-parent\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362507 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-systemd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.362511 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362523 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-node-log\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362552 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362565 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysconfig\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.362625 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.862603962 +0000 UTC m=+1.812558885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362647 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-cnibin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-system-cni-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362707 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.362761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362759 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362816 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-slash\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-log-socket\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362877 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-k8s-cni-cncf-io\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362872 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362924 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362932 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-run\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362951 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362973 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.362995 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363012 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-socket-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363029 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363071 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-sys\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363077 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-registration-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363088 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363114 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67-host-slash\") pod \"iptables-alerter-9qwxg\" (UID: \"921d8f4a-1e5d-4c4e-a4fe-94b920bf9b67\") " pod="openshift-network-operator/iptables-alerter-9qwxg" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363109 5036 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363163 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363165 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-dbus\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-systemd\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363165 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-lib-modules\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363223 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363191 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6509794-be77-4085-aeec-92819c3c9041-host\") pod \"node-ca-pgsq6\" (UID: \"d6509794-be77-4085-aeec-92819c3c9041\") " pod="openshift-image-registry/node-ca-pgsq6" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363300 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363332 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.363363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363363 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363424 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363448 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363457 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363464 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363469 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363482 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363505 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363514 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-etc-kubernetes\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363521 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-bin\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363523 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.863507274 +0000 UTC m=+1.813462240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-hostroot\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363546 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-multus-certs\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363574 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.863557641 +0000 UTC m=+1.813512569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363575 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-sys-fs\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363604 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363643 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363666 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363691 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363713 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363756 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-bin\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363764 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363775 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-sysctl-conf\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.363842 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.863829525 +0000 UTC m=+1.813784484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : configmap references non-existent config key: service-ca.crt Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363873 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-kubernetes\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364078 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364099 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92a37659-0e94-43e8-adb2-94ec2dfe09a1-kubelet-config\") pod \"global-pull-secret-syncer-nd669\" (UID: \"92a37659-0e94-43e8-adb2-94ec2dfe09a1\") " pod="kube-system/global-pull-secret-syncer-nd669" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.363991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-cni-netd\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364122 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.364159 5036 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364124 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-var-lib-kubelet\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.364202 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.864188504 +0000 UTC m=+1.814143465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : secret "metrics-daemon-secret" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364217 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364303 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-run-netns\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364311 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364343 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364368 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364406 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364431 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364433 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-run-ovn\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364442 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-multus-conf-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364465 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364491 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364499 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-etc-openvswitch\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364535 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-system-cni-dir\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364559 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-host-run-netns\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.364580 5036 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364599 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c533f37-a794-4b53-8779-86fbbb312bf6-hosts-file\") pod \"node-resolver-h7xwb\" (UID: \"5c533f37-a794-4b53-8779-86fbbb312bf6\") " pod="openshift-dns/node-resolver-h7xwb" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364610 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d-systemd-units\") pod \"ovnkube-node-zfdzh\" (UID: \"10c3ccd9-9679-41a3-ba1b-c00cfd7f2a2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364615 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.364657 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.864642557 +0000 UTC m=+1.814597483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.364711 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364721 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364747 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364772 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.364752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364824 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-cni-multus\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364848 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-host\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0777e96-1263-40e3-ad73-55e2a8eaa539-host-var-lib-kubelet\") pod \"multus-hk8sl\" (UID: \"e0777e96-1263-40e3-ad73-55e2a8eaa539\") " pod="openshift-multus/multus-hk8sl" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364921 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364961 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9251c254-07fe-47f7-9ca3-a0e00bec385c-cnibin\") pod \"multus-additional-cni-plugins-7pq9d\" (UID: \"9251c254-07fe-47f7-9ca3-a0e00bec385c\") " pod="openshift-multus/multus-additional-cni-plugins-7pq9d" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-etc-selinux\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.364998 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da7bf2d6-e040-4695-aeb3-f0bb15293c98-device-dir\") pod \"aws-ebs-csi-driver-node-rddr5\" (UID: \"da7bf2d6-e040-4695-aeb3-f0bb15293c98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rddr5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.365025 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:37.865009132 +0000 UTC m=+1.814964060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.365070 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74027051-12b8-4ebf-8e06-a9783a3c232c-etc-modprobe-d\") pod \"tuned-slpv5\" (UID: \"74027051-12b8-4ebf-8e06-a9783a3c232c\") " pod="openshift-cluster-node-tuning-operator/tuned-slpv5" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.366025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dac1afb9-e804-406f-9064-8fe8e976e0c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gfljx\" (UID: \"dac1afb9-e804-406f-9064-8fe8e976e0c6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.366162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4763a8d1-c6f0-49eb-81e3-a6919c5f4a85-metrics-tls\") pod \"dns-default-hm8lm\" (UID: \"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85\") " pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.366597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e33c44f7-69f3-4f6e-bbfd-760ba6440f2c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-92mlp\" (UID: \"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.367233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.366755 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb903f58-35c6-449e-8350-3382d4494692-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xmhg9\" (UID: \"fb903f58-35c6-449e-8350-3382d4494692\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.496738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.496710 5036 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:37.496738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.496733 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" Oct 30 21:39:37.501002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.500985 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" Oct 30 21:39:37.501072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.501036 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" Oct 30 21:39:37.507960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.507941 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:37.528879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.528856 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:37.814183 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.814115 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9"] Oct 30 21:39:37.850490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.850473 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:39:37.861513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.861488 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx"] Oct 30 21:39:37.866141 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:37.866103 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac1afb9_e804_406f_9064_8fe8e976e0c6.slice/crio-6b15e35c49a157c25d99863a2094254985cba1a616d04de9248c2656ec707ac6 WatchSource:0}: Error finding container 6b15e35c49a157c25d99863a2094254985cba1a616d04de9248c2656ec707ac6: Status 404 returned error can't find the container with id 6b15e35c49a157c25d99863a2094254985cba1a616d04de9248c2656ec707ac6 Oct 30 21:39:37.868610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:37.868700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868628 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.868700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868651 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:37.868700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868682 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868714 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868745 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868763 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.868795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868799 5036 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868816 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868799214 +0000 UTC m=+2.818754156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:37.868847 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868822 5036 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868866 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868849378 +0000 UTC m=+2.818804326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : secret "metrics-daemon-secret" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868866 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868883 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868886 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868873337 +0000 UTC m=+2.818828268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868923 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868912919 +0000 UTC m=+2.818867848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868927 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868944 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868970 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868956025 +0000 UTC m=+2.818910963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:37.869161 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:37.868990 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:38.868979195 +0000 UTC m=+2.818934124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:37.869467 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.869293 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-service-ca-bundle\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:37.874390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.874371 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:39:37.899112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.899059 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm8lm"] Oct 30 21:39:37.903111 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:37.903087 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4763a8d1_c6f0_49eb_81e3_a6919c5f4a85.slice/crio-e4db170201ca2074b625d2e098d8d3938d975fc0d03ec7dc50fababe2eef5228 WatchSource:0}: Error finding container e4db170201ca2074b625d2e098d8d3938d975fc0d03ec7dc50fababe2eef5228: Status 404 returned error can't find the container with id e4db170201ca2074b625d2e098d8d3938d975fc0d03ec7dc50fababe2eef5228 Oct 30 21:39:37.924064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:37.924040 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-92mlp"] Oct 30 21:39:37.927845 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:37.927825 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33c44f7_69f3_4f6e_bbfd_760ba6440f2c.slice/crio-2936a71ca12a8eba6afec391984ee01c1b958ef03920040d4e55a490e18661b1 WatchSource:0}: Error finding container 2936a71ca12a8eba6afec391984ee01c1b958ef03920040d4e55a490e18661b1: Status 404 returned error can't find the container with id 2936a71ca12a8eba6afec391984ee01c1b958ef03920040d4e55a490e18661b1 Oct 30 21:39:38.252170 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.252142 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:39:38.252637 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.252617 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/1.log" Oct 30 21:39:38.252700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.252658 5036 generic.go:356] "Generic (PLEG): container finished" podID="9829f19b-65f8-4435-a942-0c997ee268c3" containerID="65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f" exitCode=255 Oct 30 21:39:38.252750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.252728 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" event={"ID":"9829f19b-65f8-4435-a942-0c997ee268c3","Type":"ContainerDied","Data":"65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f"} Oct 30 21:39:38.252800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.252780 5036 scope.go:122] "RemoveContainer" containerID="dd2195bbb4028bfffb02574548f154b6faf6c0ffa02fe2b18985ad875837f77f" Oct 30 21:39:38.253126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.253110 5036 scope.go:122] "RemoveContainer" containerID="65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f" Oct 30 21:39:38.253416 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.253395 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:38.254151 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.254128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" event={"ID":"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c","Type":"ContainerStarted","Data":"2936a71ca12a8eba6afec391984ee01c1b958ef03920040d4e55a490e18661b1"} Oct 30 21:39:38.255370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.255349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" event={"ID":"fb903f58-35c6-449e-8350-3382d4494692","Type":"ContainerStarted","Data":"853e627a20b670b6e7a89f08a2606f8f7785c31af3a013afeec896dc086289a5"} Oct 30 21:39:38.256632 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.256609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" event={"ID":"dac1afb9-e804-406f-9064-8fe8e976e0c6","Type":"ContainerStarted","Data":"6b15e35c49a157c25d99863a2094254985cba1a616d04de9248c2656ec707ac6"} Oct 30 21:39:38.257888 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.257867 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm8lm" event={"ID":"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85","Type":"ContainerStarted","Data":"e4db170201ca2074b625d2e098d8d3938d975fc0d03ec7dc50fababe2eef5228"} Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885002 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885136 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885184 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:38.885254 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.885521 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.885589 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.885569729 +0000 UTC m=+4.835524671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886083 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886139 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.886123087 +0000 UTC m=+4.836078025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886199 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886210 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d5dcc446-9v622: secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886242 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls podName:adcfff7f-94e9-4cb3-b800-acdc636274ab nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.886231578 +0000 UTC m=+4.836186517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls") pod "image-registry-76d5dcc446-9v622" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab") : secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886325 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886336 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-875fcbc87-fc57r: secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886369 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls podName:528f7e4a-14f0-4e90-819a-118a0ea2c058 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.886357924 +0000 UTC m=+4.836312850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls") pod "image-registry-875fcbc87-fc57r" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058") : secret "image-registry-tls" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886418 5036 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886447 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs podName:02acb0f7-5c31-4397-ae32-fe076ef673e0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.886437602 +0000 UTC m=+4.836392529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs") pod "network-metrics-daemon-xcg97" (UID: "02acb0f7-5c31-4397-ae32-fe076ef673e0") : secret "metrics-daemon-secret" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886483 5036 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:38.886549 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:38.886507 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.886500453 +0000 UTC m=+4.836455377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:39.268258 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:39.268190 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:39:40.273794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.273765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-92mlp" event={"ID":"e33c44f7-69f3-4f6e-bbfd-760ba6440f2c","Type":"ContainerStarted","Data":"f273ce64ee246f1b93b8ca47d7c1918c57bf6d73965fc877b33949f20b8cfc03"} Oct 30 21:39:40.276233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.276187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" event={"ID":"fb903f58-35c6-449e-8350-3382d4494692","Type":"ContainerStarted","Data":"248a498de4636af93a688af22b00e433e98f62e6b161e085406210232a077b5b"} Oct 30 21:39:40.276374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.276293 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xmhg9" event={"ID":"fb903f58-35c6-449e-8350-3382d4494692","Type":"ContainerStarted","Data":"b5d7406abf0265a911d87bf3c6d3a24f82896dc2084c6311ce8dcfeb47249fc7"} Oct 30 21:39:40.277807 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.277783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gfljx" event={"ID":"dac1afb9-e804-406f-9064-8fe8e976e0c6","Type":"ContainerStarted","Data":"ae5e3c113457e25158b5ed827e28b63c8879b64bf71953aa46582dba0c736e99"} Oct 30 21:39:40.904200 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904160 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904217 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904302 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904310 5036 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:40.904424 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904387 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:44.904368153 +0000 UTC m=+8.854323096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : configmap "kube-rbac-proxy" not found Oct 30 21:39:40.904726 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.904428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:40.904726 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904435 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:39:40.904726 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904499 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs podName:a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c nodeName:}" failed. No retries permitted until 2026-10-30 21:39:44.904481457 +0000 UTC m=+8.854436399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs") pod "router-default-7df45874d9-82hkt" (UID: "a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c") : secret "router-metrics-certs-default" not found Oct 30 21:39:40.904726 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904561 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:40.904726 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:40.904598 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:44.904586266 +0000 UTC m=+8.854541237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:40.906616 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.906597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02acb0f7-5c31-4397-ae32-fe076ef673e0-metrics-certs\") pod \"network-metrics-daemon-xcg97\" (UID: \"02acb0f7-5c31-4397-ae32-fe076ef673e0\") " pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:40.906795 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.906776 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"image-registry-875fcbc87-fc57r\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:40.906852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:40.906779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"image-registry-76d5dcc446-9v622\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:41.098178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.098111 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:41.100914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.100889 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcg97" Oct 30 21:39:41.107924 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.107907 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:41.282736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.282707 5036 generic.go:356] "Generic (PLEG): container finished" podID="6b673e62-070f-4a1d-969c-1c4958789ba9" containerID="849c568fa1be6f498d38f0ac869ea137028b70f2c5fa05f71d15b74437b6ab46" exitCode=0 Oct 30 21:39:41.283191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.282779 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wh95c" event={"ID":"6b673e62-070f-4a1d-969c-1c4958789ba9","Type":"ContainerDied","Data":"849c568fa1be6f498d38f0ac869ea137028b70f2c5fa05f71d15b74437b6ab46"} Oct 30 21:39:41.283191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.283148 5036 scope.go:122] "RemoveContainer" containerID="849c568fa1be6f498d38f0ac869ea137028b70f2c5fa05f71d15b74437b6ab46" Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.289538 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm8lm" event={"ID":"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85","Type":"ContainerStarted","Data":"4bc4edf9df537376dc535ebf2247a297077ed9cefd8fb0dda6938804fc9b921d"} Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.289620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm8lm" event={"ID":"4763a8d1-c6f0-49eb-81e3-a6919c5f4a85","Type":"ContainerStarted","Data":"11992f71f4304ba25084cf73f4c87edc1c324504a52e44ff165a598e132cbdf4"} Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.289655 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.291174 5036 generic.go:356] "Generic (PLEG): container finished" podID="cc38fec3-2436-4e9e-aea8-4bf2fdab438e" containerID="e4afcd323be01722656e722fadbf2984aed59dbd602abbe37b302de9f669a686" exitCode=0 Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.291284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" event={"ID":"cc38fec3-2436-4e9e-aea8-4bf2fdab438e","Type":"ContainerDied","Data":"e4afcd323be01722656e722fadbf2984aed59dbd602abbe37b302de9f669a686"} Oct 30 21:39:41.292528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.291781 5036 scope.go:122] "RemoveContainer" containerID="e4afcd323be01722656e722fadbf2984aed59dbd602abbe37b302de9f669a686" Oct 30 21:39:41.383250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.383224 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xcg97"] Oct 30 21:39:41.386750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.386723 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:39:41.389549 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:41.389526 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02acb0f7_5c31_4397_ae32_fe076ef673e0.slice/crio-b336f3c568c11ee029f5c3a4162baffac45f8a6c68f31be647a3ac83669cbbf3 WatchSource:0}: Error finding container b336f3c568c11ee029f5c3a4162baffac45f8a6c68f31be647a3ac83669cbbf3: Status 404 returned error can't find the container with id b336f3c568c11ee029f5c3a4162baffac45f8a6c68f31be647a3ac83669cbbf3 Oct 30 21:39:41.390605 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:41.390586 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcfff7f_94e9_4cb3_b800_acdc636274ab.slice/crio-2a3e320a542ba4dd487268bf6f7a58ac218ee920491c48c96cb959837035fed6 WatchSource:0}: Error finding container 2a3e320a542ba4dd487268bf6f7a58ac218ee920491c48c96cb959837035fed6: Status 404 returned error can't find the container with id 2a3e320a542ba4dd487268bf6f7a58ac218ee920491c48c96cb959837035fed6 Oct 30 21:39:41.394759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:41.394742 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:39:41.402605 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:41.402582 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528f7e4a_14f0_4e90_819a_118a0ea2c058.slice/crio-75e9cd2b209387f95fd33334185d7ffcf5cd64d2fac4531a9777a2858656110f WatchSource:0}: Error finding container 75e9cd2b209387f95fd33334185d7ffcf5cd64d2fac4531a9777a2858656110f: Status 404 returned error can't find the container with id 75e9cd2b209387f95fd33334185d7ffcf5cd64d2fac4531a9777a2858656110f Oct 30 21:39:42.297004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.296970 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" event={"ID":"528f7e4a-14f0-4e90-819a-118a0ea2c058","Type":"ContainerStarted","Data":"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62"} Oct 30 21:39:42.297511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.297033 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" event={"ID":"528f7e4a-14f0-4e90-819a-118a0ea2c058","Type":"ContainerStarted","Data":"75e9cd2b209387f95fd33334185d7ffcf5cd64d2fac4531a9777a2858656110f"} Oct 30 21:39:42.298667 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.298640 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" event={"ID":"adcfff7f-94e9-4cb3-b800-acdc636274ab","Type":"ContainerStarted","Data":"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4"} Oct 30 21:39:42.298784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.298694 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" event={"ID":"adcfff7f-94e9-4cb3-b800-acdc636274ab","Type":"ContainerStarted","Data":"2a3e320a542ba4dd487268bf6f7a58ac218ee920491c48c96cb959837035fed6"} Oct 30 21:39:42.300622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.300573 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wh95c" event={"ID":"6b673e62-070f-4a1d-969c-1c4958789ba9","Type":"ContainerStarted","Data":"7508322662d4f4bc4275fc2ae4f0ed1fc8f64c10cea0c9346ebd0f55a19826d4"} Oct 30 21:39:42.301811 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.301784 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcg97" event={"ID":"02acb0f7-5c31-4397-ae32-fe076ef673e0","Type":"ContainerStarted","Data":"b336f3c568c11ee029f5c3a4162baffac45f8a6c68f31be647a3ac83669cbbf3"} Oct 30 21:39:42.303543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.303522 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ltt4p" event={"ID":"cc38fec3-2436-4e9e-aea8-4bf2fdab438e","Type":"ContainerStarted","Data":"b6bc8fbe830fba026142437f9d3c0d12d6571459e45af31229b10db2afaba0b9"} Oct 30 21:39:42.603009 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:42.602937 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:39:43.314728 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:43.314700 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:43.315143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:43.314699 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcg97" event={"ID":"02acb0f7-5c31-4397-ae32-fe076ef673e0","Type":"ContainerStarted","Data":"3710fcf1b104a925c046e644b6a2831b13eaec3d0be2fb8b736ce57ce71d4dfa"} Oct 30 21:39:43.315143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:43.314887 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcg97" event={"ID":"02acb0f7-5c31-4397-ae32-fe076ef673e0","Type":"ContainerStarted","Data":"87874c3a755dc46aff99a823f3544b211513568ffbf43a30df8dd42b24e31ed1"} Oct 30 21:39:44.202830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.202790 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:39:44.940337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.940295 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:44.940767 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.940348 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:44.940767 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.940389 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:44.940767 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:44.940593 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:39:44.940767 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:44.940689 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls podName:fc51b84b-d859-4ec3-97ea-ec66bb8a8756 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:52.940669489 +0000 UTC m=+16.890624429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqzzl" (UID: "fc51b84b-d859-4ec3-97ea-ec66bb8a8756") : secret "insights-runtime-extractor-tls" not found Oct 30 21:39:44.940938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.940803 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:44.942779 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.942750 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c-metrics-certs\") pod \"router-default-7df45874d9-82hkt\" (UID: \"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c\") " pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:44.998674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:44.998642 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:45.189516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.189490 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:45.189758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.189740 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:45.279077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.279004 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df45874d9-82hkt"] Oct 30 21:39:45.282665 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:45.282621 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98b79c6_cbd7_4d8a_ae09_4ecce2fdd98c.slice/crio-ea21c09f5ce3c975566b8ea35022430875eca5d07de3f9a21c5d0aabad5ce7e6 WatchSource:0}: Error finding container ea21c09f5ce3c975566b8ea35022430875eca5d07de3f9a21c5d0aabad5ce7e6: Status 404 returned error can't find the container with id ea21c09f5ce3c975566b8ea35022430875eca5d07de3f9a21c5d0aabad5ce7e6 Oct 30 21:39:45.324923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.324895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df45874d9-82hkt" event={"ID":"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c","Type":"ContainerStarted","Data":"ea21c09f5ce3c975566b8ea35022430875eca5d07de3f9a21c5d0aabad5ce7e6"} Oct 30 21:39:45.418431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.418384 5036 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:45.418761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.418746 5036 scope.go:122] "RemoveContainer" containerID="65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f" Oct 30 21:39:45.418950 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:45.418930 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:45.697622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:45.697578 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:46.330323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:46.330281 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df45874d9-82hkt" event={"ID":"a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c","Type":"ContainerStarted","Data":"70089b2b53d8be28cdab467cec298a85760b1587a0eeadb2bf4491920eadc8bd"} Oct 30 21:39:46.330785 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:46.330732 5036 scope.go:122] "RemoveContainer" containerID="65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f" Oct 30 21:39:46.330946 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:39:46.330922 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dljwb_openshift-console-operator(9829f19b-65f8-4435-a942-0c997ee268c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" podUID="9829f19b-65f8-4435-a942-0c997ee268c3" Oct 30 21:39:46.495309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:46.495281 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:46.998980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:46.998942 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:47.001664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:47.001641 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:47.335947 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:47.335877 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7df45874d9-82hkt" Oct 30 21:39:48.181025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:48.180994 5036 scope.go:122] "RemoveContainer" containerID="65029422d71857c012bb1c3b3d2e3bc4d76de4426701770ec26ee0ac6d2a6b2f" Oct 30 21:39:48.340624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:48.340596 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:39:48.341098 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:48.340717 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" event={"ID":"9829f19b-65f8-4435-a942-0c997ee268c3","Type":"ContainerStarted","Data":"b10716629c6dcad2e7579bf703a9455b2646d12292f24aebbc999b5718905975"} Oct 30 21:39:48.341098 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:48.341084 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:48.602567 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:48.602478 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dljwb" Oct 30 21:39:50.561764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:50.561733 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:39:51.305373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:51.305348 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hm8lm" Oct 30 21:39:52.375533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:52.375461 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfnx9" Oct 30 21:39:53.009089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:53.009056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:53.011534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:53.011503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc51b84b-d859-4ec3-97ea-ec66bb8a8756-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqzzl\" (UID: \"fc51b84b-d859-4ec3-97ea-ec66bb8a8756\") " pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:53.112081 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:53.112051 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqzzl" Oct 30 21:39:53.390929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:53.390901 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqzzl"] Oct 30 21:39:53.394057 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:39:53.394033 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc51b84b_d859_4ec3_97ea_ec66bb8a8756.slice/crio-c4d738cb5d5a29346a4a9b24f546766033857f957a3d81f720719df21a00c93b WatchSource:0}: Error finding container c4d738cb5d5a29346a4a9b24f546766033857f957a3d81f720719df21a00c93b: Status 404 returned error can't find the container with id c4d738cb5d5a29346a4a9b24f546766033857f957a3d81f720719df21a00c93b Oct 30 21:39:54.365146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:54.365116 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqzzl" event={"ID":"fc51b84b-d859-4ec3-97ea-ec66bb8a8756","Type":"ContainerStarted","Data":"163e9cd9e1161e78c4e76434fb693c5e174989480794324a6a89fef2167857a4"} Oct 30 21:39:54.365330 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:54.365169 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqzzl" event={"ID":"fc51b84b-d859-4ec3-97ea-ec66bb8a8756","Type":"ContainerStarted","Data":"594d1da5b94fa86f0f51e28b9b6aa6db576e0837eec0a96f0532da20c38e75cd"} Oct 30 21:39:54.365330 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:54.365188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqzzl" event={"ID":"fc51b84b-d859-4ec3-97ea-ec66bb8a8756","Type":"ContainerStarted","Data":"c4d738cb5d5a29346a4a9b24f546766033857f957a3d81f720719df21a00c93b"} Oct 30 21:39:56.375429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:56.375389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqzzl" event={"ID":"fc51b84b-d859-4ec3-97ea-ec66bb8a8756","Type":"ContainerStarted","Data":"a08b0db6ff6fd3a0cd0b716c250860bb69414cc8efb6de6bc30434e9217e934f"} Oct 30 21:39:56.676076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:56.676053 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm8lm_4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/dns/0.log" Oct 30 21:39:56.876717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:56.876688 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm8lm_4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/kube-rbac-proxy/0.log" Oct 30 21:39:57.275896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:57.275869 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h7xwb_5c533f37-a794-4b53-8779-86fbbb312bf6/dns-node-resolver/0.log" Oct 30 21:39:57.653565 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:57.653536 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zfdzh" Oct 30 21:39:57.877318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:57.877279 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76d5dcc446-9v622_adcfff7f-94e9-4cb3-b800-acdc636274ab/registry/0.log" Oct 30 21:39:58.076462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:58.076389 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-875fcbc87-fc57r_528f7e4a-14f0-4e90-819a-118a0ea2c058/registry/0.log" Oct 30 21:39:58.675722 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:58.675694 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgsq6_d6509794-be77-4085-aeec-92819c3c9041/node-ca/0.log" Oct 30 21:39:58.876625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:58.876600 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df45874d9-82hkt_a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/router/0.log" Oct 30 21:39:59.276835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:39:59.276809 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxkw8_119335ed-3944-46c6-aacd-e63aff054eb8/serve-healthcheck-canary/0.log" Oct 30 21:40:00.567558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:00.567531 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:40:04.022192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.022149 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzvcg"] Oct 30 21:40:04.054279 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.054247 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.111292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.111243 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:40:04.111292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.111243 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:40:04.111292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.111247 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:40:04.111538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.111459 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:40:04.111595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.111551 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9f5cm\"" Oct 30 21:40:04.204853 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.204824 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-wtmp\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.204853 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.204854 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcmh\" (UniqueName: \"kubernetes.io/projected/1d4d1732-7920-4436-bb20-7c243aef4029-kube-api-access-hmcmh\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.204878 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-textfile\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.204990 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-tls\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.205023 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-metrics-client-ca\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.205040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-root\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.205067 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.205084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.205370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.205191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-sys\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.305981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.305912 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.305981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.305946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-sys\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.305981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.305972 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-wtmp\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.305990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcmh\" (UniqueName: \"kubernetes.io/projected/1d4d1732-7920-4436-bb20-7c243aef4029-kube-api-access-hmcmh\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306017 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-textfile\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306049 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-tls\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306070 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-metrics-client-ca\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306085 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-root\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-sys\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306111 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-wtmp\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306228 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d4d1732-7920-4436-bb20-7c243aef4029-root\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-textfile\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.306825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.306761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d4d1732-7920-4436-bb20-7c243aef4029-metrics-client-ca\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.308434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.308404 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.308657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.308639 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d4d1732-7920-4436-bb20-7c243aef4029-node-exporter-tls\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.318152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.318131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcmh\" (UniqueName: \"kubernetes.io/projected/1d4d1732-7920-4436-bb20-7c243aef4029-kube-api-access-hmcmh\") pod \"node-exporter-mzvcg\" (UID: \"1d4d1732-7920-4436-bb20-7c243aef4029\") " pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.322902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.322883 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:40:04.401404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:04.401375 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzvcg" Oct 30 21:40:04.445280 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:40:04.445232 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4d1732_7920_4436_bb20_7c243aef4029.slice/crio-d6f02a829a9c9e6cb024199bbed12fa4077c1ace90d7b1ad8471f3c22103d4da WatchSource:0}: Error finding container d6f02a829a9c9e6cb024199bbed12fa4077c1ace90d7b1ad8471f3c22103d4da: Status 404 returned error can't find the container with id d6f02a829a9c9e6cb024199bbed12fa4077c1ace90d7b1ad8471f3c22103d4da Oct 30 21:40:05.412627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:05.412587 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvcg" event={"ID":"1d4d1732-7920-4436-bb20-7c243aef4029","Type":"ContainerStarted","Data":"d6f02a829a9c9e6cb024199bbed12fa4077c1ace90d7b1ad8471f3c22103d4da"} Oct 30 21:40:06.422195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:06.422165 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d4d1732-7920-4436-bb20-7c243aef4029" containerID="4853c7b43e90bd0771a1b061ffad8e3e7a699079f0afe5489460b934728c95e7" exitCode=0 Oct 30 21:40:06.422709 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:06.422236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvcg" event={"ID":"1d4d1732-7920-4436-bb20-7c243aef4029","Type":"ContainerDied","Data":"4853c7b43e90bd0771a1b061ffad8e3e7a699079f0afe5489460b934728c95e7"} Oct 30 21:40:07.429253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:07.429212 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvcg" event={"ID":"1d4d1732-7920-4436-bb20-7c243aef4029","Type":"ContainerStarted","Data":"195d6e5eb8c3dc4e2a990040382888e07588b956f27a2df5e385a8ccaae0c942"} Oct 30 21:40:07.429650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:07.429334 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvcg" event={"ID":"1d4d1732-7920-4436-bb20-7c243aef4029","Type":"ContainerStarted","Data":"32b9a1725069e65c3324eac09b391fa535a798a5b0050abcb719ed93bc6c55d5"} Oct 30 21:40:07.454400 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:07.454332 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mzvcg" podStartSLOduration=2.95632158 podStartE2EDuration="4.454317565s" totalImagesPullingTime="1.412050472s" totalInitContainerRuntime="85.945513ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:07.453011985 +0000 UTC m=+31.402966931" watchObservedRunningTime="2026-10-30 21:40:07.454317565 +0000 UTC m=+31.404272534" Oct 30 21:40:08.261246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:08.261218 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dr5t5" Oct 30 21:40:15.084860 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:15.084823 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:40:15.590223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:15.590177 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" podUID="adcfff7f-94e9-4cb3-b800-acdc636274ab" containerName="registry" containerID="cri-o://7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4" gracePeriod=30 Oct 30 21:40:15.864868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:15.864844 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:40:16.003995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.003960 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004049 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004073 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004091 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004117 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004141 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004521 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004173 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") pod \"adcfff7f-94e9-4cb3-b800-acdc636274ab\" (UID: \"adcfff7f-94e9-4cb3-b800-acdc636274ab\") " Oct 30 21:40:16.004712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.004652 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:16.005287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.005239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:16.006788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.006749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:16.006903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.006813 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:16.007244 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.007222 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:16.007329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.007220 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:16.007329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.007294 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "kube-api-access-26gjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:16.011485 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.011462 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted" pod "adcfff7f-94e9-4cb3-b800-acdc636274ab" (UID: "adcfff7f-94e9-4cb3-b800-acdc636274ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:16.105366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105310 5036 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-bound-sa-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105335 5036 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-image-registry-private-configuration\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105345 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-trusted-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105354 5036 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adcfff7f-94e9-4cb3-b800-acdc636274ab-installation-pull-secrets\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105365 5036 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-certificates\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105774 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105374 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gjk\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-kube-api-access-26gjk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105774 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105382 5036 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adcfff7f-94e9-4cb3-b800-acdc636274ab-registry-tls\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.105774 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.105390 5036 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adcfff7f-94e9-4cb3-b800-acdc636274ab-ca-trust-extracted\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:16.465953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.465919 5036 generic.go:356] "Generic (PLEG): container finished" podID="adcfff7f-94e9-4cb3-b800-acdc636274ab" containerID="7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4" exitCode=0 Oct 30 21:40:16.466137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.465973 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" event={"ID":"adcfff7f-94e9-4cb3-b800-acdc636274ab","Type":"ContainerDied","Data":"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4"} Oct 30 21:40:16.466137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.465990 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" Oct 30 21:40:16.466137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.466014 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d5dcc446-9v622" event={"ID":"adcfff7f-94e9-4cb3-b800-acdc636274ab","Type":"ContainerDied","Data":"2a3e320a542ba4dd487268bf6f7a58ac218ee920491c48c96cb959837035fed6"} Oct 30 21:40:16.466137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.466030 5036 scope.go:122] "RemoveContainer" containerID="7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4" Oct 30 21:40:16.474653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.474638 5036 scope.go:122] "RemoveContainer" containerID="7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4" Oct 30 21:40:16.474938 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:40:16.474919 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4\": container with ID starting with 7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4 not found: ID does not exist" containerID="7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4" Oct 30 21:40:16.474983 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.474950 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4"} err="failed to get container status \"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4\": rpc error: code = NotFound desc = could not find container \"7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4\": container with ID starting with 7e16d7b4addcc242fddb07f98665b70cced4b8381b59652907eeae31d11672f4 not found: ID does not exist" Oct 30 21:40:16.488395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.488374 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:40:16.497277 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:16.497240 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76d5dcc446-9v622"] Oct 30 21:40:18.097443 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:40:18.097404 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:40:18.185162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:18.185132 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcfff7f-94e9-4cb3-b800-acdc636274ab" path="/var/lib/kubelet/pods/adcfff7f-94e9-4cb3-b800-acdc636274ab/volumes" Oct 30 21:40:23.421912 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.421879 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:40:23.422349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.422079 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" podUID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" containerName="insights-gathering" containerID="cri-o://8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3" gracePeriod=30 Oct 30 21:40:23.684496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.684443 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:40:23.766830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.766801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") pod \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " Oct 30 21:40:23.766984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.766848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") pod \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " Oct 30 21:40:23.766984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.766867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") pod \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " Oct 30 21:40:23.766984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.766923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") pod \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\" (UID: \"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191\") " Oct 30 21:40:23.767216 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.767195 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path" pod "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" (UID: "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:23.767257 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.767224 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle" pod "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" (UID: "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:23.767257 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.767235 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle" pod "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" (UID: "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:23.769393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.769370 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk" pod "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" (UID: "e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191"). InnerVolumeSpecName "kube-api-access-jz6wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:23.867815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.867784 5036 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-service-ca-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.867815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.867812 5036 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-archives-path\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.867815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.867822 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-trusted-ca-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.868025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:23.867831 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz6wk\" (UniqueName: \"kubernetes.io/projected/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191-kube-api-access-jz6wk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:24.499960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.499870 5036 generic.go:356] "Generic (PLEG): container finished" podID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" containerID="8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3" exitCode=2 Oct 30 21:40:24.499960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.499912 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" Oct 30 21:40:24.499960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.499919 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" event={"ID":"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191","Type":"ContainerDied","Data":"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3"} Oct 30 21:40:24.499960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.499958 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lbx6-g9gk4" event={"ID":"e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191","Type":"ContainerDied","Data":"cafbd56f7d5a6e255efd7cad05bc4ce91951415b5d3a6a5e75dc8aead9accb9b"} Oct 30 21:40:24.500509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.499978 5036 scope.go:122] "RemoveContainer" containerID="8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3" Oct 30 21:40:24.508335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.508317 5036 scope.go:122] "RemoveContainer" containerID="8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3" Oct 30 21:40:24.508615 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:40:24.508599 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3\": container with ID starting with 8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3 not found: ID does not exist" containerID="8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3" Oct 30 21:40:24.508663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.508625 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3"} err="failed to get container status \"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3\": rpc error: code = NotFound desc = could not find container \"8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3\": container with ID starting with 8f410d6642d8871ecdf4964d3de116416c10572f80f1993bd4124d9511e8f9c3 not found: ID does not exist" Oct 30 21:40:24.514929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.514909 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:40:24.523362 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:24.523337 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4lbx6-g9gk4"] Oct 30 21:40:26.184917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:26.184886 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" path="/var/lib/kubelet/pods/e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191/volumes" Oct 30 21:40:40.110399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.110328 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" podUID="528f7e4a-14f0-4e90-819a-118a0ea2c058" containerName="registry" containerID="cri-o://7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62" gracePeriod=30 Oct 30 21:40:40.361577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.361521 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:40:40.502651 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502620 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502674 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502699 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502743 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502763 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502787 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.502828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502819 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.503128 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.502852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") pod \"528f7e4a-14f0-4e90-819a-118a0ea2c058\" (UID: \"528f7e4a-14f0-4e90-819a-118a0ea2c058\") " Oct 30 21:40:40.503357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.503326 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:40.503450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.503385 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:40.505588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.505539 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:40.505700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.505623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "kube-api-access-wbznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:40.505790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.505761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:40.505901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.505838 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:40.505947 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.505914 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:40.511591 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.511553 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted" pod "528f7e4a-14f0-4e90-819a-118a0ea2c058" (UID: "528f7e4a-14f0-4e90-819a-118a0ea2c058"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:40.569571 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.569542 5036 generic.go:356] "Generic (PLEG): container finished" podID="528f7e4a-14f0-4e90-819a-118a0ea2c058" containerID="7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62" exitCode=0 Oct 30 21:40:40.569702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.569598 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" Oct 30 21:40:40.569702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.569630 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" event={"ID":"528f7e4a-14f0-4e90-819a-118a0ea2c058","Type":"ContainerDied","Data":"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62"} Oct 30 21:40:40.569702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.569696 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-875fcbc87-fc57r" event={"ID":"528f7e4a-14f0-4e90-819a-118a0ea2c058","Type":"ContainerDied","Data":"75e9cd2b209387f95fd33334185d7ffcf5cd64d2fac4531a9777a2858656110f"} Oct 30 21:40:40.569887 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.569723 5036 scope.go:122] "RemoveContainer" containerID="7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62" Oct 30 21:40:40.578334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.578320 5036 scope.go:122] "RemoveContainer" containerID="7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62" Oct 30 21:40:40.578612 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:40:40.578591 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62\": container with ID starting with 7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62 not found: ID does not exist" containerID="7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62" Oct 30 21:40:40.578700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.578620 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62"} err="failed to get container status \"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62\": rpc error: code = NotFound desc = could not find container \"7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62\": container with ID starting with 7fc42a64575ec0a885aa482a20b4cc24f41d8cd5f4ed600242788eb6e9827c62 not found: ID does not exist" Oct 30 21:40:40.593314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.593294 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:40:40.600520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.600457 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-875fcbc87-fc57r"] Oct 30 21:40:40.603960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.603939 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbznk\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-kube-api-access-wbznk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.603963 5036 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-tls\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.603979 5036 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528f7e4a-14f0-4e90-819a-118a0ea2c058-ca-trust-extracted\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.603992 5036 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-registry-certificates\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.604008 5036 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-image-registry-private-configuration\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.604023 5036 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528f7e4a-14f0-4e90-819a-118a0ea2c058-bound-sa-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.604038 5036 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528f7e4a-14f0-4e90-819a-118a0ea2c058-installation-pull-secrets\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.604066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:40.604053 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528f7e4a-14f0-4e90-819a-118a0ea2c058-trusted-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:42.186403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:42.186370 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528f7e4a-14f0-4e90-819a-118a0ea2c058" path="/var/lib/kubelet/pods/528f7e4a-14f0-4e90-819a-118a0ea2c058/volumes" Oct 30 21:40:48.097075 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:40:48.097038 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:40:56.633060 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:56.633020 5036 generic.go:356] "Generic (PLEG): container finished" podID="c6c881db-f097-4bf2-af04-3aa394bdfc44" containerID="d22dbee888f12bfa6e9b845b3c0eaf45b5f3d5cea4d57bc8793297c648bf7497" exitCode=0 Oct 30 21:40:56.633490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:56.633086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" event={"ID":"c6c881db-f097-4bf2-af04-3aa394bdfc44","Type":"ContainerDied","Data":"d22dbee888f12bfa6e9b845b3c0eaf45b5f3d5cea4d57bc8793297c648bf7497"} Oct 30 21:40:56.633490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:56.633430 5036 scope.go:122] "RemoveContainer" containerID="d22dbee888f12bfa6e9b845b3c0eaf45b5f3d5cea4d57bc8793297c648bf7497" Oct 30 21:40:57.638611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:40:57.638578 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ttjhw" event={"ID":"c6c881db-f097-4bf2-af04-3aa394bdfc44","Type":"ContainerStarted","Data":"98d5aec8ddae4d4c19b71e84efdb1e40681781bc8a0399571be68481877c9b89"} Oct 30 21:41:18.097564 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:41:18.097525 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:41:48.098076 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:41:48.097969 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:42:18.097823 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:42:18.097787 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:42:48.096806 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:42:48.096766 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:43:18.097445 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:43:18.097360 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:43:48.098335 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:43:48.098298 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:43:50.210397 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210358 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6"] Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210678 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" containerName="insights-gathering" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210691 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" containerName="insights-gathering" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210708 5036 cpu_manager.go:414] "containerMap: removing container" podUID="528f7e4a-14f0-4e90-819a-118a0ea2c058" containerName="registry" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210713 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528f7e4a-14f0-4e90-819a-118a0ea2c058" containerName="registry" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210722 5036 cpu_manager.go:414] "containerMap: removing container" podUID="adcfff7f-94e9-4cb3-b800-acdc636274ab" containerName="registry" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210728 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcfff7f-94e9-4cb3-b800-acdc636274ab" containerName="registry" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210788 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d27cdf-3a8f-4dbb-aa41-98c99aa0f191" containerName="insights-gathering" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210798 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="528f7e4a-14f0-4e90-819a-118a0ea2c058" containerName="registry" Oct 30 21:43:50.210903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.210804 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcfff7f-94e9-4cb3-b800-acdc636274ab" containerName="registry" Oct 30 21:43:50.213886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.213868 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.217199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.217175 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:43:50.217345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.217215 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:43:50.217345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.217220 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kjk2g\"" Oct 30 21:43:50.223639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.223615 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6"] Oct 30 21:43:50.334247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.334211 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.334442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.334256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsqt\" (UniqueName: \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.334442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.334390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.434977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.434944 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.435155 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.434998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.435155 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.435123 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsqt\" (UniqueName: \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.435394 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.435369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.435437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.435412 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.447328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.447303 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsqt\" (UniqueName: \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.531648 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.531573 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:43:50.811003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:50.810923 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6"] Oct 30 21:43:50.814883 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:50.814851 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b2b086_5b15_4665_a08e_aa7aca562cc5.slice/crio-0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a WatchSource:0}: Error finding container 0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a: Status 404 returned error can't find the container with id 0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a Oct 30 21:43:51.294215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.294177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerStarted","Data":"0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a"} Oct 30 21:43:51.667239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.667206 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr"] Oct 30 21:43:51.671193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.671168 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.674079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.674045 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-59qnw\"" Oct 30 21:43:51.680441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.680417 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr"] Oct 30 21:43:51.773401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.773367 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh"] Oct 30 21:43:51.777163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.777135 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:51.781050 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.781019 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-xql6m\"" Oct 30 21:43:51.787152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.787120 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh"] Oct 30 21:43:51.846992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.846958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.847177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.847001 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lrb\" (UniqueName: \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.847177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.847109 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.863083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.863058 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls"] Oct 30 21:43:51.866539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.866520 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:51.870697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.870676 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wrfrs\"" Oct 30 21:43:51.883137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.883070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls"] Oct 30 21:43:51.948308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:51.948308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948238 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.948539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lrb\" (UniqueName: \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.948539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948379 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:51.948539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948450 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8z5\" (UniqueName: \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:51.948704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.948704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.948930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.948909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.959926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.959898 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588"] Oct 30 21:43:51.962015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.961991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lrb\" (UniqueName: \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:51.963851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.963833 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:51.967092 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.967055 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9sr9f\"" Oct 30 21:43:51.973995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.973971 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588"] Oct 30 21:43:51.990728 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:51.990700 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:43:52.049934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.049910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.050062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.049962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.050062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050003 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdm2m\" (UniqueName: \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.050062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.050382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050079 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.050382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050111 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtfk\" (UniqueName: \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.050382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050141 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8z5\" (UniqueName: \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.050382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050196 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.050382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.050736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.050709 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.051047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.051015 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.065862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.063371 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8z5\" (UniqueName: \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.097200 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.097166 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.152847 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.152898 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.152941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.152979 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdm2m\" (UniqueName: \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.153009 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.153065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.153050 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtfk\" (UniqueName: \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.154306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.153807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.154306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.154107 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.154627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.154459 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.154901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.154877 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.166403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.166378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtfk\" (UniqueName: \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.166518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.166378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdm2m\" (UniqueName: \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.184579 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.184536 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:43:52.278175 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.278147 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr"] Oct 30 21:43:52.282174 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:52.282148 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fda131a_475f_4410_a165_8f6574fb807c.slice/crio-fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96 WatchSource:0}: Error finding container fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96: Status 404 returned error can't find the container with id fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96 Oct 30 21:43:52.287953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.287932 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:43:52.299492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.299471 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" event={"ID":"4fda131a-475f-4410-a165-8f6574fb807c","Type":"ContainerStarted","Data":"fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96"} Oct 30 21:43:52.381498 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.381467 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh"] Oct 30 21:43:52.388479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:52.388445 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e60e79b_5986_4cfe_a90f_4c042c90bc84.slice/crio-4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8 WatchSource:0}: Error finding container 4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8: Status 404 returned error can't find the container with id 4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8 Oct 30 21:43:52.466732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.466700 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls"] Oct 30 21:43:52.470951 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:52.470922 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e1e6b2_64f6_4a5e_bbf2_2c477bbe36f7.slice/crio-1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75 WatchSource:0}: Error finding container 1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75: Status 404 returned error can't find the container with id 1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75 Oct 30 21:43:52.573476 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:52.573390 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588"] Oct 30 21:43:52.577257 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:52.577220 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c553db_b127_4594_b690_3cc82d42d505.slice/crio-4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c WatchSource:0}: Error finding container 4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c: Status 404 returned error can't find the container with id 4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c Oct 30 21:43:53.310041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.309994 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerStarted","Data":"4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c"} Oct 30 21:43:53.318338 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.318227 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" event={"ID":"5e60e79b-5986-4cfe-a90f-4c042c90bc84","Type":"ContainerStarted","Data":"4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8"} Oct 30 21:43:53.324674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.324592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" event={"ID":"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7","Type":"ContainerStarted","Data":"1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75"} Oct 30 21:43:53.363048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.361416 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn"] Oct 30 21:43:53.365812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.365243 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.371390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.370784 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mz5df\"" Oct 30 21:43:53.375798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.375775 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn"] Oct 30 21:43:53.463800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.463764 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l"] Oct 30 21:43:53.467961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.467935 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.471169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.471137 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.471297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.471214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45zb\" (UniqueName: \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.471297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.471255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.482250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.482210 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kx6b6\"" Oct 30 21:43:53.482425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.482300 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:43:53.486458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.484182 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l"] Oct 30 21:43:53.567932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.567311 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq"] Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572823 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45zb\" (UniqueName: \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572929 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572969 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.572995 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573019 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573048 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lhx\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-kube-api-access-g2lhx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573134 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573157 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.573183 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.574465 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.574858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.574676 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.583173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.583077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq"] Oct 30 21:43:53.583322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.583195 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.600197 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.599717 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45zb\" (UniqueName: \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.600197 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.600158 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-g4ztv\"" Oct 30 21:43:53.674164 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674124 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674189 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674213 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674240 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lhx\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-kube-api-access-g2lhx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674295 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674483 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674615 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.674771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.674718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.675573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.675458 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.675573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.675469 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.675754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.675674 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.676055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.676021 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.676055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.676035 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.680041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.679578 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.680603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.680524 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.687375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.687355 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lhx\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-kube-api-access-g2lhx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.692503 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.692287 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:43:53.693067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.693008 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9b6de7a-8d31-4f0e-990b-81525b0912d8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-5bm7l\" (UID: \"b9b6de7a-8d31-4f0e-990b-81525b0912d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775555 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775598 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wdm\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-kube-api-access-j6wdm\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775680 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775718 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775741 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775773 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841d6863-c461-4be1-b810-eaeeb3977fba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775799 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841d6863-c461-4be1-b810-eaeeb3977fba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.776072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.775873 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.799259 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.799227 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:43:53.876860 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.876822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.876883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.876927 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.876985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wdm\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-kube-api-access-j6wdm\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.877017 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.877060 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.877086 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.877121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841d6863-c461-4be1-b810-eaeeb3977fba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.877299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.877151 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841d6863-c461-4be1-b810-eaeeb3977fba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.878446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.878415 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.878707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.878683 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.879284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.879237 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.879631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.879608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.880894 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.880834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841d6863-c461-4be1-b810-eaeeb3977fba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.882350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.882282 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841d6863-c461-4be1-b810-eaeeb3977fba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.895736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.895714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841d6863-c461-4be1-b810-eaeeb3977fba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.900092 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.900067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.906280 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.906239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wdm\" (UniqueName: \"kubernetes.io/projected/841d6863-c461-4be1-b810-eaeeb3977fba-kube-api-access-j6wdm\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq\" (UID: \"841d6863-c461-4be1-b810-eaeeb3977fba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:53.913133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:53.913110 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:43:54.029448 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.029409 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn"] Oct 30 21:43:54.123597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.123462 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l"] Oct 30 21:43:54.230019 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.229982 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq"] Oct 30 21:43:54.251254 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:43:54.251218 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841d6863_c461_4be1_b810_eaeeb3977fba.slice/crio-9ba2675c177304c2d4531903734eafff80b7df532d381362ebd910a82b5db2ea WatchSource:0}: Error finding container 9ba2675c177304c2d4531903734eafff80b7df532d381362ebd910a82b5db2ea: Status 404 returned error can't find the container with id 9ba2675c177304c2d4531903734eafff80b7df532d381362ebd910a82b5db2ea Oct 30 21:43:54.335667 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.335629 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" event={"ID":"b9b6de7a-8d31-4f0e-990b-81525b0912d8","Type":"ContainerStarted","Data":"739963691ff15c1a4c77550529c06fd8b666cfd0b1687f68a09900cdc373f59a"} Oct 30 21:43:54.338220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.338143 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" event={"ID":"841d6863-c461-4be1-b810-eaeeb3977fba","Type":"ContainerStarted","Data":"9ba2675c177304c2d4531903734eafff80b7df532d381362ebd910a82b5db2ea"} Oct 30 21:43:54.340310 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:43:54.340208 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" event={"ID":"766350d8-1ecb-49bd-a6e2-81fe8d49683b","Type":"ContainerStarted","Data":"8880881d5d37e2387b1bda2b9a14eecfd7b604e4bb5a97e6b22ed0cb4079d3ab"} Oct 30 21:44:02.193176 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.193124 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.193678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.193230 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.193678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.193305 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.421180 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.420453 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" event={"ID":"841d6863-c461-4be1-b810-eaeeb3977fba","Type":"ContainerStarted","Data":"e8b54357ef8a3d675c4925c73cc30939b5daa449f63e59185fb30b5fec11fc2e"} Oct 30 21:44:02.631899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.631848 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.632051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.631950 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.632051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.631996 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:02.913693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:02.913649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:44:03.435245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.435120 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" event={"ID":"b9b6de7a-8d31-4f0e-990b-81525b0912d8","Type":"ContainerStarted","Data":"2d5d216674ee2514216ec915e852dd7d157db009f34803e5e5611e14cbe14393"} Oct 30 21:44:03.458590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.458522 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podStartSLOduration=1.962452926 podStartE2EDuration="10.45850389s" totalImagesPullingTime="8.496050964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:03.458048203 +0000 UTC m=+267.408003152" watchObservedRunningTime="2026-10-30 21:44:03.45850389 +0000 UTC m=+267.408458839" Oct 30 21:44:03.459063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.459026 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" podStartSLOduration=2.521805025 podStartE2EDuration="10.459015615s" totalImagesPullingTime="7.93721059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:02.446840391 +0000 UTC m=+266.396795338" watchObservedRunningTime="2026-10-30 21:44:03.459015615 +0000 UTC m=+267.408970557" Oct 30 21:44:03.798870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.798835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:44:03.805119 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.805086 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:44:03.920610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:03.920576 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:44:04.446521 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.444339 5036 generic.go:356] "Generic (PLEG): container finished" podID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerID="c04a5eff25d46a3c5b5281fe50fd37e3f097accd03a6fee810f31517ba96dd38" exitCode=0 Oct 30 21:44:04.446521 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.444443 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" event={"ID":"766350d8-1ecb-49bd-a6e2-81fe8d49683b","Type":"ContainerDied","Data":"c04a5eff25d46a3c5b5281fe50fd37e3f097accd03a6fee810f31517ba96dd38"} Oct 30 21:44:04.451276 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.449623 5036 generic.go:356] "Generic (PLEG): container finished" podID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerID="8a037cfdc5e65e9a14ed07e1dfb742daa7e32838869d7a121ffd416fa8e35599" exitCode=0 Oct 30 21:44:04.451276 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.450198 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerDied","Data":"8a037cfdc5e65e9a14ed07e1dfb742daa7e32838869d7a121ffd416fa8e35599"} Oct 30 21:44:04.451276 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.450905 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:44:04.451276 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.450950 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:44:04.452346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.452093 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:44:04.452346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:04.452295 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq" Oct 30 21:44:06.462468 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:06.462436 5036 generic.go:356] "Generic (PLEG): container finished" podID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerID="505851fc9ae732c6184e30150f3f4a482495c9939ff280aee9bb06d1b567197c" exitCode=0 Oct 30 21:44:06.462967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:06.462520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" event={"ID":"766350d8-1ecb-49bd-a6e2-81fe8d49683b","Type":"ContainerDied","Data":"505851fc9ae732c6184e30150f3f4a482495c9939ff280aee9bb06d1b567197c"} Oct 30 21:44:06.464032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:06.464012 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fda131a-475f-4410-a165-8f6574fb807c" containerID="5d512adeea93b84b41c7e81242ddc648bda463392b23488bc3cb50c0c1e2eb83" exitCode=0 Oct 30 21:44:06.464140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:06.464096 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" event={"ID":"4fda131a-475f-4410-a165-8f6574fb807c","Type":"ContainerDied","Data":"5d512adeea93b84b41c7e81242ddc648bda463392b23488bc3cb50c0c1e2eb83"} Oct 30 21:44:07.471832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.471789 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerStarted","Data":"906880326ab7770c507ab39f241191c1427be6bb022d9510a0c1909741bf7447"} Oct 30 21:44:07.473905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.473689 5036 generic.go:356] "Generic (PLEG): container finished" podID="67c553db-b127-4594-b690-3cc82d42d505" containerID="11875d3a44f8a2d5d0ef6e6437f069e9a0cb1f988f1f754262a4a5a1239e356b" exitCode=0 Oct 30 21:44:07.473905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.473770 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerDied","Data":"11875d3a44f8a2d5d0ef6e6437f069e9a0cb1f988f1f754262a4a5a1239e356b"} Oct 30 21:44:07.476775 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.476674 5036 generic.go:356] "Generic (PLEG): container finished" podID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerID="23c6b642b63a83d42ad4057783e6823f57deb33f2a51912501b2bac9e02478ab" exitCode=0 Oct 30 21:44:07.476856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.476765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" event={"ID":"5e60e79b-5986-4cfe-a90f-4c042c90bc84","Type":"ContainerDied","Data":"23c6b642b63a83d42ad4057783e6823f57deb33f2a51912501b2bac9e02478ab"} Oct 30 21:44:07.480474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.480455 5036 generic.go:356] "Generic (PLEG): container finished" podID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerID="5e9edc03445b9cf713caa7a19128eb3f7174460c1f0765db2126815eaf4fc88b" exitCode=0 Oct 30 21:44:07.480585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:07.480492 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" event={"ID":"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7","Type":"ContainerDied","Data":"5e9edc03445b9cf713caa7a19128eb3f7174460c1f0765db2126815eaf4fc88b"} Oct 30 21:44:08.488053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.488024 5036 generic.go:356] "Generic (PLEG): container finished" podID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerID="19513e248afa4789a23d108bafcd7fcc638f3cd53ecaad8651b87532970a73cb" exitCode=0 Oct 30 21:44:08.488415 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.488242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" event={"ID":"5e60e79b-5986-4cfe-a90f-4c042c90bc84","Type":"ContainerDied","Data":"19513e248afa4789a23d108bafcd7fcc638f3cd53ecaad8651b87532970a73cb"} Oct 30 21:44:08.492254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.491391 5036 generic.go:356] "Generic (PLEG): container finished" podID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerID="906880326ab7770c507ab39f241191c1427be6bb022d9510a0c1909741bf7447" exitCode=0 Oct 30 21:44:08.492254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.491637 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerDied","Data":"906880326ab7770c507ab39f241191c1427be6bb022d9510a0c1909741bf7447"} Oct 30 21:44:08.495191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.495168 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fda131a-475f-4410-a165-8f6574fb807c" containerID="7557fbca6f252b1c52b9ecc10d51f9c80bfad2bce2f6f7293963167a6586d9b2" exitCode=0 Oct 30 21:44:08.495311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:08.495198 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" event={"ID":"4fda131a-475f-4410-a165-8f6574fb807c","Type":"ContainerDied","Data":"7557fbca6f252b1c52b9ecc10d51f9c80bfad2bce2f6f7293963167a6586d9b2"} Oct 30 21:44:09.507596 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:09.507558 5036 generic.go:356] "Generic (PLEG): container finished" podID="67c553db-b127-4594-b690-3cc82d42d505" containerID="9a20ef57266fca26a57967c7a845c107a112e4f8fa024bbbbae06fe265cd7167" exitCode=0 Oct 30 21:44:09.508107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:09.507676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerDied","Data":"9a20ef57266fca26a57967c7a845c107a112e4f8fa024bbbbae06fe265cd7167"} Oct 30 21:44:09.511977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:09.511863 5036 generic.go:356] "Generic (PLEG): container finished" podID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerID="2834ccbb9de57b4069fbab60b5a17403be18aad5bb16202f2b43f4a3cbf379cd" exitCode=0 Oct 30 21:44:09.511977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:09.511929 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" event={"ID":"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7","Type":"ContainerDied","Data":"2834ccbb9de57b4069fbab60b5a17403be18aad5bb16202f2b43f4a3cbf379cd"} Oct 30 21:44:17.580420 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:17.580389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerStarted","Data":"957d2cfe95577cc8bde18f502164efa7b348d6b444750ecb4de0c6b3f88774e7"} Oct 30 21:44:17.587533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:17.587504 5036 generic.go:356] "Generic (PLEG): container finished" podID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerID="a341670f097feb6e68ca0f918c6d0da216486d23b903cb7fbed3795b0881757d" exitCode=0 Oct 30 21:44:17.587641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:17.587592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" event={"ID":"766350d8-1ecb-49bd-a6e2-81fe8d49683b","Type":"ContainerDied","Data":"a341670f097feb6e68ca0f918c6d0da216486d23b903cb7fbed3795b0881757d"} Oct 30 21:44:17.610487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:17.606621 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" podStartSLOduration=3.2832284290000002 podStartE2EDuration="26.606602091s" totalImagesPullingTime="23.323373662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:17.603135545 +0000 UTC m=+281.553090492" watchObservedRunningTime="2026-10-30 21:44:17.606602091 +0000 UTC m=+281.556557038" Oct 30 21:44:18.103986 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:44:18.103923 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:44:18.596334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.596298 5036 generic.go:356] "Generic (PLEG): container finished" podID="67c553db-b127-4594-b690-3cc82d42d505" containerID="957d2cfe95577cc8bde18f502164efa7b348d6b444750ecb4de0c6b3f88774e7" exitCode=0 Oct 30 21:44:18.596818 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.596423 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerDied","Data":"957d2cfe95577cc8bde18f502164efa7b348d6b444750ecb4de0c6b3f88774e7"} Oct 30 21:44:18.600003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.599977 5036 generic.go:356] "Generic (PLEG): container finished" podID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerID="8c86a3500d51efc125decf59c4619ec42bc64f5df24b9488f6a895bfcdcc9150" exitCode=0 Oct 30 21:44:18.600392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.600350 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" event={"ID":"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7","Type":"ContainerDied","Data":"8c86a3500d51efc125decf59c4619ec42bc64f5df24b9488f6a895bfcdcc9150"} Oct 30 21:44:18.788252 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.788218 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:44:18.841296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.841245 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\" (UniqueName: \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\") pod \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " Oct 30 21:44:18.841482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.841329 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\") pod \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " Oct 30 21:44:18.841482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.841381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\") pod \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\" (UID: \"766350d8-1ecb-49bd-a6e2-81fe8d49683b\") " Oct 30 21:44:18.842528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.842500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle" pod "766350d8-1ecb-49bd-a6e2-81fe8d49683b" (UID: "766350d8-1ecb-49bd-a6e2-81fe8d49683b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:18.847895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.847817 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util" pod "766350d8-1ecb-49bd-a6e2-81fe8d49683b" (UID: "766350d8-1ecb-49bd-a6e2-81fe8d49683b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:18.850735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.850707 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb" pod "766350d8-1ecb-49bd-a6e2-81fe8d49683b" (UID: "766350d8-1ecb-49bd-a6e2-81fe8d49683b"). InnerVolumeSpecName "kube-api-access-g45zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:18.942131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.942091 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45zb\" (UniqueName: \"kubernetes.io/projected/766350d8-1ecb-49bd-a6e2-81fe8d49683b-kube-api-access-g45zb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:18.942131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.942131 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:18.942394 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:18.942146 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/766350d8-1ecb-49bd-a6e2-81fe8d49683b-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:19.608776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.608651 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" Oct 30 21:44:19.608776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.608697 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lc8jn" event={"ID":"766350d8-1ecb-49bd-a6e2-81fe8d49683b","Type":"ContainerDied","Data":"8880881d5d37e2387b1bda2b9a14eecfd7b604e4bb5a97e6b22ed0cb4079d3ab"} Oct 30 21:44:19.608776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.608738 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8880881d5d37e2387b1bda2b9a14eecfd7b604e4bb5a97e6b22ed0cb4079d3ab" Oct 30 21:44:19.818311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.817943 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:44:19.840696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.840672 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:44:19.965799 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965762 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\") pod \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " Oct 30 21:44:19.965967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965812 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\") pod \"67c553db-b127-4594-b690-3cc82d42d505\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " Oct 30 21:44:19.965967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\") pod \"67c553db-b127-4594-b690-3cc82d42d505\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " Oct 30 21:44:19.965967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\") pod \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " Oct 30 21:44:19.965967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965945 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\" (UniqueName: \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\") pod \"67c553db-b127-4594-b690-3cc82d42d505\" (UID: \"67c553db-b127-4594-b690-3cc82d42d505\") " Oct 30 21:44:19.966189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.965987 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\" (UniqueName: \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\") pod \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\" (UID: \"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7\") " Oct 30 21:44:19.967166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.967114 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle" pod "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" (UID: "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:19.967522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.967495 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle" pod "67c553db-b127-4594-b690-3cc82d42d505" (UID: "67c553db-b127-4594-b690-3cc82d42d505"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:19.972500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.972471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util" pod "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" (UID: "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:19.972658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.972491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util" pod "67c553db-b127-4594-b690-3cc82d42d505" (UID: "67c553db-b127-4594-b690-3cc82d42d505"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:19.974707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.974683 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk" pod "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" (UID: "d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7"). InnerVolumeSpecName "kube-api-access-9wtfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:19.978360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:19.978315 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m" pod "67c553db-b127-4594-b690-3cc82d42d505" (UID: "67c553db-b127-4594-b690-3cc82d42d505"). InnerVolumeSpecName "kube-api-access-cdm2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:20.066789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066750 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtfk\" (UniqueName: \"kubernetes.io/projected/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-kube-api-access-9wtfk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.066789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066784 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.066789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066795 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.067023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066803 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c553db-b127-4594-b690-3cc82d42d505-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.067023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066811 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.067023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.066820 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdm2m\" (UniqueName: \"kubernetes.io/projected/67c553db-b127-4594-b690-3cc82d42d505-kube-api-access-cdm2m\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:20.615813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.615787 5036 generic.go:356] "Generic (PLEG): container finished" podID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerID="822ebac9998eeab9fe040a173e096d1f5e163b9f8fc57627b5637388dfc771f0" exitCode=0 Oct 30 21:44:20.616161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.615867 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerDied","Data":"822ebac9998eeab9fe040a173e096d1f5e163b9f8fc57627b5637388dfc771f0"} Oct 30 21:44:20.617817 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.617798 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" Oct 30 21:44:20.617911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.617817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7p588" event={"ID":"67c553db-b127-4594-b690-3cc82d42d505","Type":"ContainerDied","Data":"4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c"} Oct 30 21:44:20.617911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.617862 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b42136fab86415e4576f17eddfae1efe8152d4e61775b48a507c088afcd2f8c" Oct 30 21:44:20.619638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.619617 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" event={"ID":"d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7","Type":"ContainerDied","Data":"1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75"} Oct 30 21:44:20.619732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.619648 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1781376794a69a9a743e818d8b9893cedefb5e80435901d1dcae615ef80d8b75" Oct 30 21:44:20.619732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:20.619664 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wtls" Oct 30 21:44:21.625857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.625819 5036 generic.go:356] "Generic (PLEG): container finished" podID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerID="3040cc1bd6a317b24f52bd228427511a598a5119898ce49e717635e9d4ec6523" exitCode=0 Oct 30 21:44:21.626312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.625904 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" event={"ID":"5e60e79b-5986-4cfe-a90f-4c042c90bc84","Type":"ContainerDied","Data":"3040cc1bd6a317b24f52bd228427511a598a5119898ce49e717635e9d4ec6523"} Oct 30 21:44:21.628011 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.627991 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fda131a-475f-4410-a165-8f6574fb807c" containerID="b4105bb9d79a48e4217e6cd4fe18f6e4be119de172533ef48bd64f78c178a0eb" exitCode=0 Oct 30 21:44:21.628109 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.628061 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" event={"ID":"4fda131a-475f-4410-a165-8f6574fb807c","Type":"ContainerDied","Data":"b4105bb9d79a48e4217e6cd4fe18f6e4be119de172533ef48bd64f78c178a0eb"} Oct 30 21:44:21.777626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.777596 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:44:21.881602 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.881578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\") pod \"16b2b086-5b15-4665-a08e-aa7aca562cc5\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " Oct 30 21:44:21.881751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.881648 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\") pod \"16b2b086-5b15-4665-a08e-aa7aca562cc5\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " Oct 30 21:44:21.881751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.881716 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\" (UniqueName: \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\") pod \"16b2b086-5b15-4665-a08e-aa7aca562cc5\" (UID: \"16b2b086-5b15-4665-a08e-aa7aca562cc5\") " Oct 30 21:44:21.882319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.882285 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle" pod "16b2b086-5b15-4665-a08e-aa7aca562cc5" (UID: "16b2b086-5b15-4665-a08e-aa7aca562cc5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:21.883873 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.883842 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt" pod "16b2b086-5b15-4665-a08e-aa7aca562cc5" (UID: "16b2b086-5b15-4665-a08e-aa7aca562cc5"). InnerVolumeSpecName "kube-api-access-wmsqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:21.886617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.886595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util" pod "16b2b086-5b15-4665-a08e-aa7aca562cc5" (UID: "16b2b086-5b15-4665-a08e-aa7aca562cc5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:21.983090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.983056 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsqt\" (UniqueName: \"kubernetes.io/projected/16b2b086-5b15-4665-a08e-aa7aca562cc5-kube-api-access-wmsqt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:21.983090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.983086 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:21.983090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:21.983095 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16b2b086-5b15-4665-a08e-aa7aca562cc5-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.634582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.634545 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" event={"ID":"16b2b086-5b15-4665-a08e-aa7aca562cc5","Type":"ContainerDied","Data":"0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a"} Oct 30 21:44:22.635028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.634604 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f044256ab2d1e5b96f9e510b92fcb9897efee82bbcc9501deb13dfc6bbcc71a" Oct 30 21:44:22.635028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.634707 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklkk6" Oct 30 21:44:22.783543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.783522 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:44:22.793926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.793908 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:44:22.889712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889636 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\") pod \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " Oct 30 21:44:22.889869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889727 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\" (UniqueName: \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\") pod \"4fda131a-475f-4410-a165-8f6574fb807c\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " Oct 30 21:44:22.889869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\") pod \"4fda131a-475f-4410-a165-8f6574fb807c\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " Oct 30 21:44:22.889869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\" (UniqueName: \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\") pod \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " Oct 30 21:44:22.889869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\") pod \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\" (UID: \"5e60e79b-5986-4cfe-a90f-4c042c90bc84\") " Oct 30 21:44:22.890069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.889871 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\") pod \"4fda131a-475f-4410-a165-8f6574fb807c\" (UID: \"4fda131a-475f-4410-a165-8f6574fb807c\") " Oct 30 21:44:22.890356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.890323 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle" pod "5e60e79b-5986-4cfe-a90f-4c042c90bc84" (UID: "5e60e79b-5986-4cfe-a90f-4c042c90bc84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:22.890473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.890380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle" pod "4fda131a-475f-4410-a165-8f6574fb807c" (UID: "4fda131a-475f-4410-a165-8f6574fb807c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:22.892117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.892089 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5" pod "5e60e79b-5986-4cfe-a90f-4c042c90bc84" (UID: "5e60e79b-5986-4cfe-a90f-4c042c90bc84"). InnerVolumeSpecName "kube-api-access-lm8z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:22.892216 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.892159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb" pod "4fda131a-475f-4410-a165-8f6574fb807c" (UID: "4fda131a-475f-4410-a165-8f6574fb807c"). InnerVolumeSpecName "kube-api-access-b4lrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:22.896028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.895996 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util" pod "5e60e79b-5986-4cfe-a90f-4c042c90bc84" (UID: "5e60e79b-5986-4cfe-a90f-4c042c90bc84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:22.897761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.897740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util" pod "4fda131a-475f-4410-a165-8f6574fb807c" (UID: "4fda131a-475f-4410-a165-8f6574fb807c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:22.991335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991309 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.991335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991334 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8z5\" (UniqueName: \"kubernetes.io/projected/5e60e79b-5986-4cfe-a90f-4c042c90bc84-kube-api-access-lm8z5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.991488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991344 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.991488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991352 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fda131a-475f-4410-a165-8f6574fb807c-util\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.991488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991361 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e60e79b-5986-4cfe-a90f-4c042c90bc84-bundle\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:22.991488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:22.991370 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4lrb\" (UniqueName: \"kubernetes.io/projected/4fda131a-475f-4410-a165-8f6574fb807c-kube-api-access-b4lrb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:23.641037 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.641003 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" Oct 30 21:44:23.641037 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.641014 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k6rnh" event={"ID":"5e60e79b-5986-4cfe-a90f-4c042c90bc84","Type":"ContainerDied","Data":"4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8"} Oct 30 21:44:23.641541 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.641098 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c78a406493e7e1bac0dbe09a68e000f7baa2fc3ee1cd18b970afd6e7dbf13a8" Oct 30 21:44:23.643060 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.643034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" event={"ID":"4fda131a-475f-4410-a165-8f6574fb807c","Type":"ContainerDied","Data":"fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96"} Oct 30 21:44:23.643060 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.643060 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vfvlr" Oct 30 21:44:23.643211 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:23.643080 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca43286f1b0a4edbb5eb4365102bd3165ec63a373bf826da2569daaf62dfc96" Oct 30 21:44:25.265225 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265197 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn"] Oct 30 21:44:25.265604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265578 5036 cpu_manager.go:414] "containerMap: removing container" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="pull" Oct 30 21:44:25.265604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265593 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="pull" Oct 30 21:44:25.265604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265603 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="util" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265608 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="util" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265616 5036 cpu_manager.go:414] "containerMap: removing container" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265622 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265630 5036 cpu_manager.go:414] "containerMap: removing container" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265636 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265645 5036 cpu_manager.go:414] "containerMap: removing container" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="util" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265649 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="util" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265660 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265665 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265672 5036 cpu_manager.go:414] "containerMap: removing container" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265676 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265685 5036 cpu_manager.go:414] "containerMap: removing container" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265689 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265697 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265701 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265707 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265713 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265720 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265724 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="pull" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265730 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="extract" Oct 30 21:44:25.265725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265734 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265743 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265747 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265752 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265756 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265765 5036 cpu_manager.go:414] "containerMap: removing container" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265772 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265782 5036 cpu_manager.go:414] "containerMap: removing container" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265787 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="util" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265793 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265797 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265804 5036 cpu_manager.go:414] "containerMap: removing container" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="pull" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265808 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="pull" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265862 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e60e79b-5986-4cfe-a90f-4c042c90bc84" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265870 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="766350d8-1ecb-49bd-a6e2-81fe8d49683b" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265875 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b2b086-5b15-4665-a08e-aa7aca562cc5" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265883 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c553db-b127-4594-b690-3cc82d42d505" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265888 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e1e6b2-64f6-4a5e-bbf2-2c477bbe36f7" containerName="extract" Oct 30 21:44:25.266314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.265892 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fda131a-475f-4410-a165-8f6574fb807c" containerName="extract" Oct 30 21:44:25.269170 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.269154 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.276945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.276925 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:44:25.280184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.280168 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwlz7\"" Oct 30 21:44:25.280579 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.280563 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:44:25.280691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.280676 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:44:25.307313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.307278 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn"] Oct 30 21:44:25.378001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.377972 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:44:25.378161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.378142 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:44:25.413072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413048 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.413186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.413186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413105 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btq5\" (UniqueName: \"kubernetes.io/projected/57deb1a9-3a2f-4e86-a1d4-9804381ff676-kube-api-access-5btq5\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.413285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413236 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.413358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413344 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.413399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.413371 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514253 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514337 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514524 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514556 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btq5\" (UniqueName: \"kubernetes.io/projected/57deb1a9-3a2f-4e86-a1d4-9804381ff676-kube-api-access-5btq5\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.514971 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.514944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.517125 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.517067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.517125 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.517067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.517281 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.517164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57deb1a9-3a2f-4e86-a1d4-9804381ff676-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.517403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.517385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57deb1a9-3a2f-4e86-a1d4-9804381ff676-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.526855 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.526829 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btq5\" (UniqueName: \"kubernetes.io/projected/57deb1a9-3a2f-4e86-a1d4-9804381ff676-kube-api-access-5btq5\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-pszxn\" (UID: \"57deb1a9-3a2f-4e86-a1d4-9804381ff676\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.583110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.583077 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:25.888215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:25.888181 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn"] Oct 30 21:44:25.892923 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:25.892898 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57deb1a9_3a2f_4e86_a1d4_9804381ff676.slice/crio-7b05db2859957c7d2c5affbf4a8ed96a5f476e196a2d23ca0bd75fc1163b1691 WatchSource:0}: Error finding container 7b05db2859957c7d2c5affbf4a8ed96a5f476e196a2d23ca0bd75fc1163b1691: Status 404 returned error can't find the container with id 7b05db2859957c7d2c5affbf4a8ed96a5f476e196a2d23ca0bd75fc1163b1691 Oct 30 21:44:26.662756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:26.662711 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" event={"ID":"57deb1a9-3a2f-4e86-a1d4-9804381ff676","Type":"ContainerStarted","Data":"7b05db2859957c7d2c5affbf4a8ed96a5f476e196a2d23ca0bd75fc1163b1691"} Oct 30 21:44:29.679641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:29.679607 5036 generic.go:356] "Generic (PLEG): container finished" podID="57deb1a9-3a2f-4e86-a1d4-9804381ff676" containerID="ad9cacb80f0d09a161cad0e5661d44ce583dd850ccb5fd11a67aedb86a1f4c59" exitCode=0 Oct 30 21:44:29.680118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:29.679691 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" event={"ID":"57deb1a9-3a2f-4e86-a1d4-9804381ff676","Type":"ContainerDied","Data":"ad9cacb80f0d09a161cad0e5661d44ce583dd850ccb5fd11a67aedb86a1f4c59"} Oct 30 21:44:30.686134 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.686085 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" event={"ID":"57deb1a9-3a2f-4e86-a1d4-9804381ff676","Type":"ContainerStarted","Data":"73c4705621c7ececde0d3c23bd5cb7ae029fdf02bbf486b05ba862c26705fcd1"} Oct 30 21:44:30.686582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.686186 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:30.750234 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.750177 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" podStartSLOduration=1.979855541 podStartE2EDuration="5.750162838s" totalImagesPullingTime="3.28762929s" totalInitContainerRuntime="482.678007ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.748546634 +0000 UTC m=+294.698501562" watchObservedRunningTime="2026-10-30 21:44:30.750162838 +0000 UTC m=+294.700117784" Oct 30 21:44:30.786863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.786832 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb"] Oct 30 21:44:30.790512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.790490 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:30.798042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.798011 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fz475\"" Oct 30 21:44:30.805664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.805639 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:44:30.805759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.805696 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:30.818515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.818489 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb"] Oct 30 21:44:30.961818 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.961730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8017426f-165a-4561-81b5-a55447f39a11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:30.961818 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:30.961763 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfvv\" (UniqueName: \"kubernetes.io/projected/8017426f-165a-4561-81b5-a55447f39a11-kube-api-access-jkfvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.063004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.062967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8017426f-165a-4561-81b5-a55447f39a11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.063004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.063005 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfvv\" (UniqueName: \"kubernetes.io/projected/8017426f-165a-4561-81b5-a55447f39a11-kube-api-access-jkfvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.063369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.063349 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8017426f-165a-4561-81b5-a55447f39a11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.075044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.075011 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfvv\" (UniqueName: \"kubernetes.io/projected/8017426f-165a-4561-81b5-a55447f39a11-kube-api-access-jkfvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb\" (UID: \"8017426f-165a-4561-81b5-a55447f39a11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.101958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.101934 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" Oct 30 21:44:31.391493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.391362 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb"] Oct 30 21:44:31.394767 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:31.394742 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8017426f_165a_4561_81b5_a55447f39a11.slice/crio-6df1e7c00d6e8b5ff1035efc061bc8a2d0001f941232a7889db2041a1068b25b WatchSource:0}: Error finding container 6df1e7c00d6e8b5ff1035efc061bc8a2d0001f941232a7889db2041a1068b25b: Status 404 returned error can't find the container with id 6df1e7c00d6e8b5ff1035efc061bc8a2d0001f941232a7889db2041a1068b25b Oct 30 21:44:31.692452 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:31.692363 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" event={"ID":"8017426f-165a-4561-81b5-a55447f39a11","Type":"ContainerStarted","Data":"6df1e7c00d6e8b5ff1035efc061bc8a2d0001f941232a7889db2041a1068b25b"} Oct 30 21:44:33.707613 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:33.707576 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" event={"ID":"8017426f-165a-4561-81b5-a55447f39a11","Type":"ContainerStarted","Data":"8411809afaf977d2f061dff6bb08f19207acf04f427272f1a1258661012ae3a5"} Oct 30 21:44:33.734085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:33.734026 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dzsbb" podStartSLOduration=1.929129032 podStartE2EDuration="3.734012945s" totalImagesPullingTime="1.804883913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:33.732249147 +0000 UTC m=+297.682204093" watchObservedRunningTime="2026-10-30 21:44:33.734012945 +0000 UTC m=+297.683967891" Oct 30 21:44:36.208166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:36.208131 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:44:36.210545 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:36.210525 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:44:41.394483 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.394448 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6"] Oct 30 21:44:41.401143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.401123 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:41.408088 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.408072 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-z4bh4\"" Oct 30 21:44:41.408698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.408674 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:44:41.408817 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.408728 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:44:41.408817 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.408746 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:44:41.409667 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.409643 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6"] Oct 30 21:44:41.455843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.455815 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzjp\" (UniqueName: \"kubernetes.io/projected/7d32db30-79b4-4ff0-89d2-4f759c9e2cc4-kube-api-access-8nzjp\") pod \"dns-operator-controller-manager-8557cbdb84-dgpz6\" (UID: \"7d32db30-79b4-4ff0-89d2-4f759c9e2cc4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:41.556530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.556498 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzjp\" (UniqueName: \"kubernetes.io/projected/7d32db30-79b4-4ff0-89d2-4f759c9e2cc4-kube-api-access-8nzjp\") pod \"dns-operator-controller-manager-8557cbdb84-dgpz6\" (UID: \"7d32db30-79b4-4ff0-89d2-4f759c9e2cc4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:41.577791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.577765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzjp\" (UniqueName: \"kubernetes.io/projected/7d32db30-79b4-4ff0-89d2-4f759c9e2cc4-kube-api-access-8nzjp\") pod \"dns-operator-controller-manager-8557cbdb84-dgpz6\" (UID: \"7d32db30-79b4-4ff0-89d2-4f759c9e2cc4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:41.694783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.694703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-pszxn" Oct 30 21:44:41.715101 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:41.715074 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:42.017713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:42.017685 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6"] Oct 30 21:44:42.020321 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:42.020291 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d32db30_79b4_4ff0_89d2_4f759c9e2cc4.slice/crio-9b119a8e41495decea784cd6c6afa12f6d87aff83174cf709788f007acf41a7f WatchSource:0}: Error finding container 9b119a8e41495decea784cd6c6afa12f6d87aff83174cf709788f007acf41a7f: Status 404 returned error can't find the container with id 9b119a8e41495decea784cd6c6afa12f6d87aff83174cf709788f007acf41a7f Oct 30 21:44:42.022650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:42.022630 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:44:42.749520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:42.749483 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" event={"ID":"7d32db30-79b4-4ff0-89d2-4f759c9e2cc4","Type":"ContainerStarted","Data":"9b119a8e41495decea784cd6c6afa12f6d87aff83174cf709788f007acf41a7f"} Oct 30 21:44:45.418339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.418306 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dqqn4"] Oct 30 21:44:45.421790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.421769 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.434802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.434779 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ksh6f\"" Oct 30 21:44:45.443292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.443245 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:44:45.443398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.443325 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:44:45.453318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.453297 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dqqn4"] Oct 30 21:44:45.487414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.487389 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-bound-sa-token\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.487531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.487417 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vkm\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-kube-api-access-d4vkm\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.587834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.587805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-bound-sa-token\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.587989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.587849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vkm\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-kube-api-access-d4vkm\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.605365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.605327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-bound-sa-token\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.605746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.605727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vkm\" (UniqueName: \"kubernetes.io/projected/a74483a5-a12e-4d32-a79d-f03a26c0d5e4-kube-api-access-d4vkm\") pod \"cert-manager-597b95cb84-dqqn4\" (UID: \"a74483a5-a12e-4d32-a79d-f03a26c0d5e4\") " pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.741804 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.741718 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dqqn4" Oct 30 21:44:45.767989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.767961 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" event={"ID":"7d32db30-79b4-4ff0-89d2-4f759c9e2cc4","Type":"ContainerStarted","Data":"6bbcc026c346b5e57dd1ed81b4902293c6f3bfe5a138082fcd49129ee610671a"} Oct 30 21:44:45.768132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.768085 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:45.795306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:45.794341 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" podStartSLOduration=1.830710294 podStartE2EDuration="4.794321793s" totalImagesPullingTime="2.963611499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:45.790887508 +0000 UTC m=+309.740842455" watchObservedRunningTime="2026-10-30 21:44:45.794321793 +0000 UTC m=+309.744276739" Oct 30 21:44:46.030022 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.029910 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dqqn4"] Oct 30 21:44:46.037111 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:46.037080 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda74483a5_a12e_4d32_a79d_f03a26c0d5e4.slice/crio-e8e1ffdc18494982852d148cfebaa64c9a77b1325ce14dad09f7a9a04296380d WatchSource:0}: Error finding container e8e1ffdc18494982852d148cfebaa64c9a77b1325ce14dad09f7a9a04296380d: Status 404 returned error can't find the container with id e8e1ffdc18494982852d148cfebaa64c9a77b1325ce14dad09f7a9a04296380d Oct 30 21:44:46.152087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.152057 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptctk"] Oct 30 21:44:46.156568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.156552 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:46.161985 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.161965 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cnfh5\"" Oct 30 21:44:46.173476 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.173455 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptctk"] Oct 30 21:44:46.293291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.293192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59qt\" (UniqueName: \"kubernetes.io/projected/0127e352-a013-4fee-bd75-e074f768c1a6-kube-api-access-s59qt\") pod \"authorino-operator-7b7bb4bc77-ptctk\" (UID: \"0127e352-a013-4fee-bd75-e074f768c1a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:46.393741 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.393709 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59qt\" (UniqueName: \"kubernetes.io/projected/0127e352-a013-4fee-bd75-e074f768c1a6-kube-api-access-s59qt\") pod \"authorino-operator-7b7bb4bc77-ptctk\" (UID: \"0127e352-a013-4fee-bd75-e074f768c1a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:46.411185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.411150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59qt\" (UniqueName: \"kubernetes.io/projected/0127e352-a013-4fee-bd75-e074f768c1a6-kube-api-access-s59qt\") pod \"authorino-operator-7b7bb4bc77-ptctk\" (UID: \"0127e352-a013-4fee-bd75-e074f768c1a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:46.469788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.469762 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:46.751611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.751580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptctk"] Oct 30 21:44:46.755172 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:46.755141 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0127e352_a013_4fee_bd75_e074f768c1a6.slice/crio-c66268673edaa0aa3e730e228f9a96508e51a03d3526ba7030c08b0fa89d5ca7 WatchSource:0}: Error finding container c66268673edaa0aa3e730e228f9a96508e51a03d3526ba7030c08b0fa89d5ca7: Status 404 returned error can't find the container with id c66268673edaa0aa3e730e228f9a96508e51a03d3526ba7030c08b0fa89d5ca7 Oct 30 21:44:46.773384 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.773354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" event={"ID":"0127e352-a013-4fee-bd75-e074f768c1a6","Type":"ContainerStarted","Data":"c66268673edaa0aa3e730e228f9a96508e51a03d3526ba7030c08b0fa89d5ca7"} Oct 30 21:44:46.774512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:46.774491 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dqqn4" event={"ID":"a74483a5-a12e-4d32-a79d-f03a26c0d5e4","Type":"ContainerStarted","Data":"e8e1ffdc18494982852d148cfebaa64c9a77b1325ce14dad09f7a9a04296380d"} Oct 30 21:44:48.115563 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:44:48.115524 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:44:48.786863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:48.786830 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" event={"ID":"0127e352-a013-4fee-bd75-e074f768c1a6","Type":"ContainerStarted","Data":"24bec2bdd0e5e301cb66c2bc53f2a85a8318f4306cc609b684088c20adc44a9a"} Oct 30 21:44:48.787076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:48.786940 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:48.811357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:48.811305 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" podStartSLOduration=0.957473591 podStartE2EDuration="2.81128938s" totalImagesPullingTime="1.853815789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:48.809724882 +0000 UTC m=+312.759679847" watchObservedRunningTime="2026-10-30 21:44:48.81128938 +0000 UTC m=+312.761244322" Oct 30 21:44:49.792661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:49.792620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dqqn4" event={"ID":"a74483a5-a12e-4d32-a79d-f03a26c0d5e4","Type":"ContainerStarted","Data":"8205ca873926fcecffa18b0234142cad61219f186c7c5a53adc0a0321b2194c9"} Oct 30 21:44:49.812689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:49.812633 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dqqn4" podStartSLOduration=1.622523346 podStartE2EDuration="4.812616894s" totalImagesPullingTime="3.190093548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:49.811311813 +0000 UTC m=+313.761266763" watchObservedRunningTime="2026-10-30 21:44:49.812616894 +0000 UTC m=+313.762571840" Oct 30 21:44:53.306617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.306579 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc"] Oct 30 21:44:53.309510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.309491 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.319519 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.319496 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v5hvq\"" Oct 30 21:44:53.328624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.328546 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc"] Oct 30 21:44:53.359296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.359250 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2add61fa-c2e9-4045-ae79-a20fb7eb3332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.359396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.359357 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfnr\" (UniqueName: \"kubernetes.io/projected/2add61fa-c2e9-4045-ae79-a20fb7eb3332-kube-api-access-ckfnr\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.460327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.460299 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2add61fa-c2e9-4045-ae79-a20fb7eb3332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.460452 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.460343 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfnr\" (UniqueName: \"kubernetes.io/projected/2add61fa-c2e9-4045-ae79-a20fb7eb3332-kube-api-access-ckfnr\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.460685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.460666 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2add61fa-c2e9-4045-ae79-a20fb7eb3332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.472195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.472173 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfnr\" (UniqueName: \"kubernetes.io/projected/2add61fa-c2e9-4045-ae79-a20fb7eb3332-kube-api-access-ckfnr\") pod \"kuadrant-operator-controller-manager-678d87766f-9mpqc\" (UID: \"2add61fa-c2e9-4045-ae79-a20fb7eb3332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.622102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.622020 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:53.912373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:53.912337 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc"] Oct 30 21:44:53.914611 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:53.914578 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2add61fa_c2e9_4045_ae79_a20fb7eb3332.slice/crio-267bdccbfeaf73faf658e1207bcdaaa2b883fcf8b9f5a75cdb767141d00a5681 WatchSource:0}: Error finding container 267bdccbfeaf73faf658e1207bcdaaa2b883fcf8b9f5a75cdb767141d00a5681: Status 404 returned error can't find the container with id 267bdccbfeaf73faf658e1207bcdaaa2b883fcf8b9f5a75cdb767141d00a5681 Oct 30 21:44:54.817423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:54.817378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" event={"ID":"2add61fa-c2e9-4045-ae79-a20fb7eb3332","Type":"ContainerStarted","Data":"267bdccbfeaf73faf658e1207bcdaaa2b883fcf8b9f5a75cdb767141d00a5681"} Oct 30 21:44:56.776970 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:56.776939 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dgpz6" Oct 30 21:44:58.841371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.841330 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" event={"ID":"2add61fa-c2e9-4045-ae79-a20fb7eb3332","Type":"ContainerStarted","Data":"3f0ebcd4d9eebd8cbf70baea993a574954780db1c7581c64cef597f98a685db8"} Oct 30 21:44:58.841759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.841417 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:44:58.889471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.889407 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" podStartSLOduration=1.805536013 podStartE2EDuration="5.889387761s" totalImagesPullingTime="4.083851748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:58.885821331 +0000 UTC m=+322.835776276" watchObservedRunningTime="2026-10-30 21:44:58.889387761 +0000 UTC m=+322.839342708" Oct 30 21:44:58.964019 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.963986 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7"] Oct 30 21:44:58.967687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.967671 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.998120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.998093 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cskcg\"" Oct 30 21:44:58.998893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.998870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.998921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21da5c9-b269-4799-88ad-5bcd9251c418-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.998994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49bk\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-kube-api-access-g49bk\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999119 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999141 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999298 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:58.999336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:58.999331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.002315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.002287 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7"] Oct 30 21:44:59.099968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.099876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21da5c9-b269-4799-88ad-5bcd9251c418-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.099968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.099927 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.099968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.099946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49bk\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-kube-api-access-g49bk\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.099978 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100004 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100026 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100119 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100460 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100572 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100733 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100658 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.100878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.100857 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.101080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.101060 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21da5c9-b269-4799-88ad-5bcd9251c418-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.102632 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.102608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.102683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.102656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.113806 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.113784 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.114105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.114088 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49bk\" (UniqueName: \"kubernetes.io/projected/e21da5c9-b269-4799-88ad-5bcd9251c418-kube-api-access-g49bk\") pod \"data-science-gateway-data-science-gateway-class-65844c896f5cvq7\" (UID: \"e21da5c9-b269-4799-88ad-5bcd9251c418\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.280578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.280545 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:44:59.575672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.575645 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7"] Oct 30 21:44:59.577644 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:44:59.577619 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21da5c9_b269_4799_88ad_5bcd9251c418.slice/crio-0f3693632e944c33e7f0470848112a6674f40d13c7d4dbcc7106b9173648d2bd WatchSource:0}: Error finding container 0f3693632e944c33e7f0470848112a6674f40d13c7d4dbcc7106b9173648d2bd: Status 404 returned error can't find the container with id 0f3693632e944c33e7f0470848112a6674f40d13c7d4dbcc7106b9173648d2bd Oct 30 21:44:59.579760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.579725 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:59.579843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.579796 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:59.579843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.579825 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:59.725494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.725461 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq"] Oct 30 21:44:59.729036 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.729015 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:44:59.731999 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.731978 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:44:59.732571 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.732553 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-np89f\"" Oct 30 21:44:59.737679 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.737658 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq"] Oct 30 21:44:59.740965 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.740259 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:44:59.794862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.794835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptctk" Oct 30 21:44:59.848615 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.848578 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" event={"ID":"e21da5c9-b269-4799-88ad-5bcd9251c418","Type":"ContainerStarted","Data":"1e19b63c08fc3f401eeac36366ed1d6e344b28feee9d0fa3491f487fab497776"} Oct 30 21:44:59.849042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.848649 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" event={"ID":"e21da5c9-b269-4799-88ad-5bcd9251c418","Type":"ContainerStarted","Data":"0f3693632e944c33e7f0470848112a6674f40d13c7d4dbcc7106b9173648d2bd"} Oct 30 21:44:59.874304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.874177 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" podStartSLOduration=1.874162369 podStartE2EDuration="1.874162369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.873086806 +0000 UTC m=+323.823041769" watchObservedRunningTime="2026-10-30 21:44:59.874162369 +0000 UTC m=+323.824117314" Oct 30 21:44:59.906585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.906552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7t5\" (UniqueName: \"kubernetes.io/projected/ee31829b-82c1-40ee-8039-980b1c76e810-kube-api-access-9s7t5\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:44:59.906727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.906609 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee31829b-82c1-40ee-8039-980b1c76e810-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:44:59.906727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:44:59.906693 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee31829b-82c1-40ee-8039-980b1c76e810-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.007224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.007196 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7t5\" (UniqueName: \"kubernetes.io/projected/ee31829b-82c1-40ee-8039-980b1c76e810-kube-api-access-9s7t5\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.007401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.007232 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee31829b-82c1-40ee-8039-980b1c76e810-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.007401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.007255 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee31829b-82c1-40ee-8039-980b1c76e810-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.008045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.008007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee31829b-82c1-40ee-8039-980b1c76e810-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.009837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.009812 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee31829b-82c1-40ee-8039-980b1c76e810-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.022659 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.022633 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7t5\" (UniqueName: \"kubernetes.io/projected/ee31829b-82c1-40ee-8039-980b1c76e810-kube-api-access-9s7t5\") pod \"kuadrant-console-plugin-66c4fb575d-gldwq\" (UID: \"ee31829b-82c1-40ee-8039-980b1c76e810\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.044314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.044253 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" Oct 30 21:45:00.281530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.281485 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:45:00.286610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.286577 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:45:00.330614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.330582 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq"] Oct 30 21:45:00.333895 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:45:00.333870 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee31829b_82c1_40ee_8039_980b1c76e810.slice/crio-fd8ea90bcd02b059ff3b77a76c0353ffb05313fbc3d064fb68fbb12f4b86b277 WatchSource:0}: Error finding container fd8ea90bcd02b059ff3b77a76c0353ffb05313fbc3d064fb68fbb12f4b86b277: Status 404 returned error can't find the container with id fd8ea90bcd02b059ff3b77a76c0353ffb05313fbc3d064fb68fbb12f4b86b277 Oct 30 21:45:00.854697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.854663 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" event={"ID":"ee31829b-82c1-40ee-8039-980b1c76e810","Type":"ContainerStarted","Data":"fd8ea90bcd02b059ff3b77a76c0353ffb05313fbc3d064fb68fbb12f4b86b277"} Oct 30 21:45:00.855176 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.854950 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:45:00.856150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:00.856129 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896f5cvq7" Oct 30 21:45:05.883718 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:05.883685 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" event={"ID":"ee31829b-82c1-40ee-8039-980b1c76e810","Type":"ContainerStarted","Data":"2a83ae4e9548b1beb65dc6b75a5e13a312d8ec118a181bb17671c63db1090c4e"} Oct 30 21:45:09.851148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:09.851121 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mpqc" Oct 30 21:45:09.873715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:09.873662 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-gldwq" podStartSLOduration=6.290161059 podStartE2EDuration="10.873648099s" totalImagesPullingTime="4.58348704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.91366428 +0000 UTC m=+329.863619227" watchObservedRunningTime="2026-10-30 21:45:09.873648099 +0000 UTC m=+333.823603058" Oct 30 21:45:18.097769 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:45:18.097732 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:45:22.991440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:22.991405 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k"] Oct 30 21:45:22.996720 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:22.996698 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.004720 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.004694 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:45:23.004830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.004700 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:45:23.004830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.004787 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:45:23.005944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.005919 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-dl7tk\"" Oct 30 21:45:23.005944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.005924 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:45:23.006055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.005925 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:45:23.007057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.007038 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k"] Oct 30 21:45:23.096631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.096596 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-metrics-certs\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.096783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.096668 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-cert\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.096783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.096762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-manager-config\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.096853 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.096800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpqm\" (UniqueName: \"kubernetes.io/projected/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-kube-api-access-hnpqm\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.197239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.197204 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpqm\" (UniqueName: \"kubernetes.io/projected/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-kube-api-access-hnpqm\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.197474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.197292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-metrics-certs\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.197474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.197372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-cert\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.197474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.197414 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-manager-config\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.198066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.198041 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-manager-config\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.200007 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.199985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-metrics-certs\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.200099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.200019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-cert\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.209626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.209606 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpqm\" (UniqueName: \"kubernetes.io/projected/2bd2d20c-9b3e-443c-b6ef-4b815908d64b-kube-api-access-hnpqm\") pod \"jobset-controller-manager-b9df85f75-p2q4k\" (UID: \"2bd2d20c-9b3e-443c-b6ef-4b815908d64b\") " pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.308830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.308759 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:23.587059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.586981 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k"] Oct 30 21:45:23.590199 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:45:23.590169 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd2d20c_9b3e_443c_b6ef_4b815908d64b.slice/crio-23d964aff095473bc9be6ddd5a07e477a824b0ebbc5b51f3984f44352f995785 WatchSource:0}: Error finding container 23d964aff095473bc9be6ddd5a07e477a824b0ebbc5b51f3984f44352f995785: Status 404 returned error can't find the container with id 23d964aff095473bc9be6ddd5a07e477a824b0ebbc5b51f3984f44352f995785 Oct 30 21:45:23.976726 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:23.976677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" event={"ID":"2bd2d20c-9b3e-443c-b6ef-4b815908d64b","Type":"ContainerStarted","Data":"23d964aff095473bc9be6ddd5a07e477a824b0ebbc5b51f3984f44352f995785"} Oct 30 21:45:26.995173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:26.995139 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" event={"ID":"2bd2d20c-9b3e-443c-b6ef-4b815908d64b","Type":"ContainerStarted","Data":"e0fda6ec56baca35e7a2994208190ef3e03274550bbfa42ac93502aea4c4df94"} Oct 30 21:45:26.995587 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:26.995213 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:27.025589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:27.025543 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" podStartSLOduration=2.136473959 podStartE2EDuration="5.025529448s" totalImagesPullingTime="2.889055489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:27.023130745 +0000 UTC m=+350.973085692" watchObservedRunningTime="2026-10-30 21:45:27.025529448 +0000 UTC m=+350.975484393" Oct 30 21:45:38.005988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:45:38.005948 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-b9df85f75-p2q4k" Oct 30 21:45:48.097622 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:45:48.097582 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:46:18.098342 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:46:18.098246 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:46:48.097383 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:46:48.097343 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:47:05.493418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:05.493368 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:05.493972 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:05.493434 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:06.501245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.501219 5036 generic.go:356] "Generic (PLEG): container finished" podID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerID="2d5d216674ee2514216ec915e852dd7d157db009f34803e5e5611e14cbe14393" exitCode=0 Oct 30 21:47:06.501603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.501254 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" event={"ID":"b9b6de7a-8d31-4f0e-990b-81525b0912d8","Type":"ContainerDied","Data":"2d5d216674ee2514216ec915e852dd7d157db009f34803e5e5611e14cbe14393"} Oct 30 21:47:06.501603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.501598 5036 scope.go:122] "RemoveContainer" containerID="2d5d216674ee2514216ec915e852dd7d157db009f34803e5e5611e14cbe14393" Oct 30 21:47:06.502738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.502707 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:06.502796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.502778 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:06.502832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:06.502812 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:07.507980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:07.507948 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" event={"ID":"b9b6de7a-8d31-4f0e-990b-81525b0912d8","Type":"ContainerStarted","Data":"9185dec90ad6ab6202d18b6b5cea0014771ed7848aba02d78b55d33f78f5164b"} Oct 30 21:47:07.798912 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:07.798825 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:47:07.799062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:07.799007 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:47:07.799102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:07.799068 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:47:08.799220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:08.799183 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:47:08.799627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:08.799253 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:47:09.799101 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:09.799062 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:47:09.799291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:09.799121 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:47:10.798963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:10.798934 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:47:10.799132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:10.798984 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:47:11.798728 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:11.798692 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:47:11.799140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:11.798745 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:47:12.802750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:12.802719 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:47:12.803194 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:12.803169 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:47:12.804012 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:12.803989 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" Oct 30 21:47:18.097765 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:47:18.097711 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:47:45.886954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.886870 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:47:45.890835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.890817 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:45.898462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.898437 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:47:45.898462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.898444 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:47:45.899106 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.899079 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:47:45.899204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.899108 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:47:45.899204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.899177 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-s65f5\"" Oct 30 21:47:45.904662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.904642 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:47:45.956657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.956628 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:45.956816 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:45.956663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzr2p\" (UniqueName: \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.058102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.058075 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.058251 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.058106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzr2p\" (UniqueName: \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.060771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.060748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.070397 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.070370 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzr2p\" (UniqueName: \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\") pod \"ai-gateway-controller-7d85675f98-jq97l\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.204091 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.204025 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:46.478955 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.478930 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:47:46.481604 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:47:46.481575 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd37c6a_6474_4905_a71d_e18ac2c5574f.slice/crio-3f35d9a9e595379449332e11b3216e43921a37bf28ef31a3378592203f7cfa5f WatchSource:0}: Error finding container 3f35d9a9e595379449332e11b3216e43921a37bf28ef31a3378592203f7cfa5f: Status 404 returned error can't find the container with id 3f35d9a9e595379449332e11b3216e43921a37bf28ef31a3378592203f7cfa5f Oct 30 21:47:46.740089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:46.740001 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" event={"ID":"cfd37c6a-6474-4905-a71d-e18ac2c5574f","Type":"ContainerStarted","Data":"3f35d9a9e595379449332e11b3216e43921a37bf28ef31a3378592203f7cfa5f"} Oct 30 21:47:48.106589 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:47:48.106552 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:47:49.593627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.593594 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9gpsn"] Oct 30 21:47:49.598825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.598799 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.607616 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.607592 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:47:49.609465 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.609440 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9gpsn"] Oct 30 21:47:49.628830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.628801 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r65q6"] Oct 30 21:47:49.633040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.633019 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.638491 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.638459 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cf9bl\"" Oct 30 21:47:49.638660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.638631 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r65q6"] Oct 30 21:47:49.692619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.692586 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed55f70-e371-4b9a-9258-e70a781e919c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.692765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.692647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxm2\" (UniqueName: \"kubernetes.io/projected/fed55f70-e371-4b9a-9258-e70a781e919c-kube-api-access-bcxm2\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.761034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.761000 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" event={"ID":"cfd37c6a-6474-4905-a71d-e18ac2c5574f","Type":"ContainerStarted","Data":"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b"} Oct 30 21:47:49.761209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.761119 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:47:49.791454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.791398 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" podStartSLOduration=2.38610729 podStartE2EDuration="4.791384082s" totalImagesPullingTime="2.405276792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.78939206 +0000 UTC m=+493.739347005" watchObservedRunningTime="2026-10-30 21:47:49.791384082 +0000 UTC m=+493.741339027" Oct 30 21:47:49.794305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.794253 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxm2\" (UniqueName: \"kubernetes.io/projected/fed55f70-e371-4b9a-9258-e70a781e919c-kube-api-access-bcxm2\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.794517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.794320 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccfc7e4-a88a-461d-9223-c1554dcaef70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.794517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.794397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthb6\" (UniqueName: \"kubernetes.io/projected/5ccfc7e4-a88a-461d-9223-c1554dcaef70-kube-api-access-mthb6\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.794698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.794587 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed55f70-e371-4b9a-9258-e70a781e919c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.795345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.795322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed55f70-e371-4b9a-9258-e70a781e919c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.808664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.808639 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxm2\" (UniqueName: \"kubernetes.io/projected/fed55f70-e371-4b9a-9258-e70a781e919c-kube-api-access-bcxm2\") pod \"payload-processing-55fbbbf9dc-9gpsn\" (UID: \"fed55f70-e371-4b9a-9258-e70a781e919c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.895937 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.895899 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthb6\" (UniqueName: \"kubernetes.io/projected/5ccfc7e4-a88a-461d-9223-c1554dcaef70-kube-api-access-mthb6\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.896409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.896031 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccfc7e4-a88a-461d-9223-c1554dcaef70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.897048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.897022 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccfc7e4-a88a-461d-9223-c1554dcaef70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.908460 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.908433 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthb6\" (UniqueName: \"kubernetes.io/projected/5ccfc7e4-a88a-461d-9223-c1554dcaef70-kube-api-access-mthb6\") pod \"payload-pre-processing-7b98bb678c-r65q6\" (UID: \"5ccfc7e4-a88a-461d-9223-c1554dcaef70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:49.914157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.914137 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:49.949044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:49.949015 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:50.195779 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:50.195570 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9gpsn"] Oct 30 21:47:50.198063 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:47:50.198031 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed55f70_e371_4b9a_9258_e70a781e919c.slice/crio-b55e2bd81cc7e63e8e5eb104be9fbfc80f5cb72f78e841ef75a3df96158ad192 WatchSource:0}: Error finding container b55e2bd81cc7e63e8e5eb104be9fbfc80f5cb72f78e841ef75a3df96158ad192: Status 404 returned error can't find the container with id b55e2bd81cc7e63e8e5eb104be9fbfc80f5cb72f78e841ef75a3df96158ad192 Oct 30 21:47:50.244441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:50.244414 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r65q6"] Oct 30 21:47:50.246249 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:47:50.246219 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc7e4_a88a_461d_9223_c1554dcaef70.slice/crio-c761e55d43f52200340fc9d4a3f7727b0a0e3eb03c72d0e3f950402f4e53ac1c WatchSource:0}: Error finding container c761e55d43f52200340fc9d4a3f7727b0a0e3eb03c72d0e3f950402f4e53ac1c: Status 404 returned error can't find the container with id c761e55d43f52200340fc9d4a3f7727b0a0e3eb03c72d0e3f950402f4e53ac1c Oct 30 21:47:50.769522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:50.769474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" event={"ID":"5ccfc7e4-a88a-461d-9223-c1554dcaef70","Type":"ContainerStarted","Data":"c761e55d43f52200340fc9d4a3f7727b0a0e3eb03c72d0e3f950402f4e53ac1c"} Oct 30 21:47:50.771604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:50.771575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" event={"ID":"fed55f70-e371-4b9a-9258-e70a781e919c","Type":"ContainerStarted","Data":"b55e2bd81cc7e63e8e5eb104be9fbfc80f5cb72f78e841ef75a3df96158ad192"} Oct 30 21:47:52.785288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.785168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" event={"ID":"fed55f70-e371-4b9a-9258-e70a781e919c","Type":"ContainerStarted","Data":"ef33493499afa4fedb232c7021516ea3d8a8f6ac59950d0f3a4c0ce5a53fa2ec"} Oct 30 21:47:52.785790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.785358 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:47:52.786868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.786845 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" event={"ID":"5ccfc7e4-a88a-461d-9223-c1554dcaef70","Type":"ContainerStarted","Data":"c0e23d196df676589ed15b225b4463a6901ee27ab72c519bf2e0ba1b4ec3e63a"} Oct 30 21:47:52.786986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.786954 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:47:52.804704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.804660 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" podStartSLOduration=1.436392453 podStartE2EDuration="3.804648018s" totalImagesPullingTime="2.368255565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:52.802312681 +0000 UTC m=+496.752267627" watchObservedRunningTime="2026-10-30 21:47:52.804648018 +0000 UTC m=+496.754602973" Oct 30 21:47:52.820398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:47:52.820173 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" podStartSLOduration=1.5330766759999999 podStartE2EDuration="3.820160534s" totalImagesPullingTime="2.287083858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:52.819082087 +0000 UTC m=+496.769037032" watchObservedRunningTime="2026-10-30 21:47:52.820160534 +0000 UTC m=+496.770115483" Oct 30 21:48:00.774185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:00.774157 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:48:03.795194 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:03.795166 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r65q6" Oct 30 21:48:03.795743 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:03.795252 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9gpsn" Oct 30 21:48:14.584070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.584036 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:48:14.584610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.584358 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" podUID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" containerName="manager" containerID="cri-o://73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b" gracePeriod=30 Oct 30 21:48:14.831655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.831631 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:48:14.896969 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.896940 5036 generic.go:356] "Generic (PLEG): container finished" podID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" containerID="73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b" exitCode=0 Oct 30 21:48:14.897124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.896992 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" Oct 30 21:48:14.897124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.897026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" event={"ID":"cfd37c6a-6474-4905-a71d-e18ac2c5574f","Type":"ContainerDied","Data":"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b"} Oct 30 21:48:14.897124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.897088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jq97l" event={"ID":"cfd37c6a-6474-4905-a71d-e18ac2c5574f","Type":"ContainerDied","Data":"3f35d9a9e595379449332e11b3216e43921a37bf28ef31a3378592203f7cfa5f"} Oct 30 21:48:14.897124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.897117 5036 scope.go:122] "RemoveContainer" containerID="73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b" Oct 30 21:48:14.905973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.905953 5036 scope.go:122] "RemoveContainer" containerID="73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b" Oct 30 21:48:14.906237 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:48:14.906217 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b\": container with ID starting with 73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b not found: ID does not exist" containerID="73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b" Oct 30 21:48:14.906331 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.906245 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b"} err="failed to get container status \"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b\": rpc error: code = NotFound desc = could not find container \"73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b\": container with ID starting with 73d5c720312c0a39bff097402e526d77bec32d6d538e2d897aa825766755044b not found: ID does not exist" Oct 30 21:48:14.912505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.912483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\" (UniqueName: \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\") pod \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " Oct 30 21:48:14.912553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.912535 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\") pod \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\" (UID: \"cfd37c6a-6474-4905-a71d-e18ac2c5574f\") " Oct 30 21:48:14.914670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.914644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p" pod "cfd37c6a-6474-4905-a71d-e18ac2c5574f" (UID: "cfd37c6a-6474-4905-a71d-e18ac2c5574f"). InnerVolumeSpecName "kube-api-access-mzr2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:14.914780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:14.914757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs" pod "cfd37c6a-6474-4905-a71d-e18ac2c5574f" (UID: "cfd37c6a-6474-4905-a71d-e18ac2c5574f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:15.014091 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:15.014053 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzr2p\" (UniqueName: \"kubernetes.io/projected/cfd37c6a-6474-4905-a71d-e18ac2c5574f-kube-api-access-mzr2p\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.014091 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:15.014086 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd37c6a-6474-4905-a71d-e18ac2c5574f-metrics-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.219401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:15.219375 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:48:15.222149 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:15.222125 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jq97l"] Oct 30 21:48:16.186186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:16.186139 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" path="/var/lib/kubelet/pods/cfd37c6a-6474-4905-a71d-e18ac2c5574f/volumes" Oct 30 21:48:18.097710 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:48:18.097668 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:48:25.139334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.139300 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2"] Oct 30 21:48:25.139792 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.139713 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" containerName="manager" Oct 30 21:48:25.139792 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.139727 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" containerName="manager" Oct 30 21:48:25.139881 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.139833 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd37c6a-6474-4905-a71d-e18ac2c5574f" containerName="manager" Oct 30 21:48:25.146973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.146955 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.155064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.155040 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:48:25.155183 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.155071 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:48:25.155183 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.155171 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:48:25.155296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.155216 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4lw49\"" Oct 30 21:48:25.155556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.155531 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2"] Oct 30 21:48:25.203563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203535 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.203763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwth\" (UniqueName: \"kubernetes.io/projected/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-kube-api-access-kdwth\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.203763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203687 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.203763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203718 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.203763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203760 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.203997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.203817 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.304908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.304876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.304908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.304910 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.304954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305023 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwth\" (UniqueName: \"kubernetes.io/projected/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-kube-api-access-kdwth\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305088 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305136 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305391 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305521 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305497 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.305593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.305569 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.307452 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.307423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.307738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.307713 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.316849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.316823 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwth\" (UniqueName: \"kubernetes.io/projected/0b9abb0b-7989-467e-86a9-41bbe7ce3a59-kube-api-access-kdwth\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mm6v2\" (UID: \"0b9abb0b-7989-467e-86a9-41bbe7ce3a59\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.461406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.461322 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:25.749247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.749102 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2"] Oct 30 21:48:25.752044 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:48:25.752011 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9abb0b_7989_467e_86a9_41bbe7ce3a59.slice/crio-757e71e57d5cbfaa3b8e29cbf5738ca6bd8cd85846a954dad63c251e88908599 WatchSource:0}: Error finding container 757e71e57d5cbfaa3b8e29cbf5738ca6bd8cd85846a954dad63c251e88908599: Status 404 returned error can't find the container with id 757e71e57d5cbfaa3b8e29cbf5738ca6bd8cd85846a954dad63c251e88908599 Oct 30 21:48:25.955894 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:25.955859 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" event={"ID":"0b9abb0b-7989-467e-86a9-41bbe7ce3a59","Type":"ContainerStarted","Data":"757e71e57d5cbfaa3b8e29cbf5738ca6bd8cd85846a954dad63c251e88908599"} Oct 30 21:48:27.260669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.260634 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n"] Oct 30 21:48:27.265102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.265077 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.275414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.275391 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:48:27.276178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.276148 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n"] Oct 30 21:48:27.328847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.328807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.329021 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.328890 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.329021 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.328939 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.329021 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.328976 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8mt\" (UniqueName: \"kubernetes.io/projected/0768611d-547a-42bd-b85b-65ae92070f3d-kube-api-access-rl8mt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.329189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.329027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0768611d-547a-42bd-b85b-65ae92070f3d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.329244 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.329192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430150 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430203 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430241 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8mt\" (UniqueName: \"kubernetes.io/projected/0768611d-547a-42bd-b85b-65ae92070f3d-kube-api-access-rl8mt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430343 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0768611d-547a-42bd-b85b-65ae92070f3d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430384 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430416 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.430957 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.430931 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.431239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.431220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.431902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.431873 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.434042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.434015 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0768611d-547a-42bd-b85b-65ae92070f3d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.434376 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.434350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0768611d-547a-42bd-b85b-65ae92070f3d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.444792 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.444758 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8mt\" (UniqueName: \"kubernetes.io/projected/0768611d-547a-42bd-b85b-65ae92070f3d-kube-api-access-rl8mt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w9r5n\" (UID: \"0768611d-547a-42bd-b85b-65ae92070f3d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.585142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.585114 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:27.867461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.867376 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n"] Oct 30 21:48:27.871550 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:48:27.871503 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0768611d_547a_42bd_b85b_65ae92070f3d.slice/crio-083cc49115430a5efcc1887d9b17fefcf69b7801e4b81b1907001a5b0a140223 WatchSource:0}: Error finding container 083cc49115430a5efcc1887d9b17fefcf69b7801e4b81b1907001a5b0a140223: Status 404 returned error can't find the container with id 083cc49115430a5efcc1887d9b17fefcf69b7801e4b81b1907001a5b0a140223 Oct 30 21:48:27.969535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.969496 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" event={"ID":"0768611d-547a-42bd-b85b-65ae92070f3d","Type":"ContainerStarted","Data":"083cc49115430a5efcc1887d9b17fefcf69b7801e4b81b1907001a5b0a140223"} Oct 30 21:48:27.971516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.971485 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" event={"ID":"0b9abb0b-7989-467e-86a9-41bbe7ce3a59","Type":"ContainerStarted","Data":"6b305adae14caf4d2aff667dd1d2a305b3eddbfb0c59f49f6e1a39fe28a19211"} Oct 30 21:48:27.993544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:27.993495 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" podStartSLOduration=1.5728464199999999 podStartE2EDuration="2.993482242s" totalImagesPullingTime="1.420635822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:27.990807127 +0000 UTC m=+531.940762073" watchObservedRunningTime="2026-10-30 21:48:27.993482242 +0000 UTC m=+531.943437188" Oct 30 21:48:28.461810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:28.461720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:28.980917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:28.980878 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" event={"ID":"0768611d-547a-42bd-b85b-65ae92070f3d","Type":"ContainerStarted","Data":"4999a5c36da47f8b99061407b1322d86f63cd16306c3972d651bf86f57b02087"} Oct 30 21:48:29.000397 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:29.000350 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" podStartSLOduration=1.719494134 podStartE2EDuration="2.00033541s" totalImagesPullingTime="280.841276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:28.999380159 +0000 UTC m=+532.949335105" watchObservedRunningTime="2026-10-30 21:48:29.00033541 +0000 UTC m=+532.950290368" Oct 30 21:48:29.585812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:29.585767 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:31.595450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.595336 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9qcwk"] Oct 30 21:48:31.601124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.601096 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.609518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.609491 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:48:31.611868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.611841 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9qcwk"] Oct 30 21:48:31.673635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673599 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkclh\" (UniqueName: \"kubernetes.io/projected/e388b1ae-8405-44ab-949b-4d0d12da7c32-kube-api-access-zkclh\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.673802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673645 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e388b1ae-8405-44ab-949b-4d0d12da7c32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.673802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.673802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.673903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673830 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-home\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.673938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.673911 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-home\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774580 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkclh\" (UniqueName: \"kubernetes.io/projected/e388b1ae-8405-44ab-949b-4d0d12da7c32-kube-api-access-zkclh\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e388b1ae-8405-44ab-949b-4d0d12da7c32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774731 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.774977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774763 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.775038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.774980 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-home\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.775100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.775031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.775161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.775102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.777167 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.777142 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e388b1ae-8405-44ab-949b-4d0d12da7c32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.777370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.777352 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e388b1ae-8405-44ab-949b-4d0d12da7c32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.788008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.787985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkclh\" (UniqueName: \"kubernetes.io/projected/e388b1ae-8405-44ab-949b-4d0d12da7c32-kube-api-access-zkclh\") pod \"echo-simulator-kserve-64b6f8f96c-9qcwk\" (UID: \"e388b1ae-8405-44ab-949b-4d0d12da7c32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:31.919336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:31.919304 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:32.201288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:32.201250 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9qcwk"] Oct 30 21:48:32.203174 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:48:32.203146 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode388b1ae_8405_44ab_949b_4d0d12da7c32.slice/crio-738e128cec4f04717ee2e4e04ddee853197cad4052f32828ed06843244c50655 WatchSource:0}: Error finding container 738e128cec4f04717ee2e4e04ddee853197cad4052f32828ed06843244c50655: Status 404 returned error can't find the container with id 738e128cec4f04717ee2e4e04ddee853197cad4052f32828ed06843244c50655 Oct 30 21:48:33.007343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:33.007248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" event={"ID":"e388b1ae-8405-44ab-949b-4d0d12da7c32","Type":"ContainerStarted","Data":"738e128cec4f04717ee2e4e04ddee853197cad4052f32828ed06843244c50655"} Oct 30 21:48:35.021599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.021565 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" event={"ID":"e388b1ae-8405-44ab-949b-4d0d12da7c32","Type":"ContainerStarted","Data":"af44e240d5758adc22bcb9666bbc82acaa9cbb5c8a34328f2c3de4d9a5c33876"} Oct 30 21:48:35.048471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.048418 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" podStartSLOduration=2.09928222 podStartE2EDuration="4.048402661s" totalImagesPullingTime="1.949120441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:35.046158226 +0000 UTC m=+538.996113173" watchObservedRunningTime="2026-10-30 21:48:35.048402661 +0000 UTC m=+538.998357606" Oct 30 21:48:35.282801 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.282730 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm"] Oct 30 21:48:35.286553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.286531 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.291870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.291845 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:48:35.301524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.301500 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm"] Oct 30 21:48:35.411796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.411763 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.411995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.411803 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7hs\" (UniqueName: \"kubernetes.io/projected/015c7202-4d5d-461c-99d1-cc3eca48e7b5-kube-api-access-jd7hs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.411995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.411930 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.412126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.412011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.412126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.412045 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.412126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.412074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.462471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.462437 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:35.481462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.481440 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:35.513417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513387 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513511 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7hs\" (UniqueName: \"kubernetes.io/projected/015c7202-4d5d-461c-99d1-cc3eca48e7b5-kube-api-access-jd7hs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513549 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513680 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513865 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.513943 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.514006 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.513979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.516037 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.516006 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015c7202-4d5d-461c-99d1-cc3eca48e7b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.516242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.516221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015c7202-4d5d-461c-99d1-cc3eca48e7b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.525380 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.525359 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7hs\" (UniqueName: \"kubernetes.io/projected/015c7202-4d5d-461c-99d1-cc3eca48e7b5-kube-api-access-jd7hs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm\" (UID: \"015c7202-4d5d-461c-99d1-cc3eca48e7b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.602369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.602253 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:35.891490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.891467 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm"] Oct 30 21:48:35.894020 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:48:35.893986 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod015c7202_4d5d_461c_99d1_cc3eca48e7b5.slice/crio-f1711497ee11daff2dfda70c976ef47bf8ba6970ade9e214066489c426654f6c WatchSource:0}: Error finding container f1711497ee11daff2dfda70c976ef47bf8ba6970ade9e214066489c426654f6c: Status 404 returned error can't find the container with id f1711497ee11daff2dfda70c976ef47bf8ba6970ade9e214066489c426654f6c Oct 30 21:48:35.920226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:35.920182 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:36.031150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:36.031100 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" event={"ID":"015c7202-4d5d-461c-99d1-cc3eca48e7b5","Type":"ContainerStarted","Data":"f1711497ee11daff2dfda70c976ef47bf8ba6970ade9e214066489c426654f6c"} Oct 30 21:48:36.050750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:36.050698 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" Oct 30 21:48:37.041077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:37.041034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" event={"ID":"015c7202-4d5d-461c-99d1-cc3eca48e7b5","Type":"ContainerStarted","Data":"988094ba5ecea259bc52cb595c15023afd64aa8e54862f6f1a6231ad922850c5"} Oct 30 21:48:37.061604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:37.061547 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podStartSLOduration=1.773223689 podStartE2EDuration="2.061529869s" totalImagesPullingTime="288.30618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:37.059639666 +0000 UTC m=+541.009594613" watchObservedRunningTime="2026-10-30 21:48:37.061529869 +0000 UTC m=+541.011484815" Oct 30 21:48:37.585946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:37.585917 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:37.599112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:37.599089 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:37.602355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:37.602333 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:38.065919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:38.065884 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" Oct 30 21:48:41.919458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:41.919421 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:41.931662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:41.931639 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:42.082658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:42.082630 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" Oct 30 21:48:45.602525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:45.602497 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:45.615055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:45.615028 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:46.105217 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:46.105191 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:48:48.097910 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:48:48.097870 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:48:53.983048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:53.983005 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl"] Oct 30 21:48:53.986906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:53.986875 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.019304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.019256 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl"] Oct 30 21:48:54.086786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.086786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086784 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf00519b-3c51-4867-bd8a-b68854743ae5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086799 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086833 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.086987 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.087030 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.087146 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj46p\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-kube-api-access-wj46p\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.087279 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.087195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188125 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188091 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj46p\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-kube-api-access-wj46p\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188133 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188184 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188208 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf00519b-3c51-4867-bd8a-b68854743ae5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188230 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188288 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188607 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188722 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.188979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.188957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.189047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.189032 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.189107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.189079 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf00519b-3c51-4867-bd8a-b68854743ae5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.190933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.190902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.191504 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.191477 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.213302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.213258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj46p\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-kube-api-access-wj46p\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.230478 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.230453 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf00519b-3c51-4867-bd8a-b68854743ae5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dl8xl\" (UID: \"bf00519b-3c51-4867-bd8a-b68854743ae5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.300896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.300805 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:54.596836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.596816 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl"] Oct 30 21:48:54.598945 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:48:54.598921 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf00519b_3c51_4867_bd8a_b68854743ae5.slice/crio-b0607b75e9752af30c5403f8d635e35589e6411ddd9fffd786a05cdc12de3000 WatchSource:0}: Error finding container b0607b75e9752af30c5403f8d635e35589e6411ddd9fffd786a05cdc12de3000: Status 404 returned error can't find the container with id b0607b75e9752af30c5403f8d635e35589e6411ddd9fffd786a05cdc12de3000 Oct 30 21:48:54.601399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.601360 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:54.601517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.601432 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:54.601517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:54.601485 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:55.145152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.145113 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" event={"ID":"bf00519b-3c51-4867-bd8a-b68854743ae5","Type":"ContainerStarted","Data":"68ae4c03648447d225c58f24da2ef497b4fa39e4406de6386c655496e7425d14"} Oct 30 21:48:55.145665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.145178 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" event={"ID":"bf00519b-3c51-4867-bd8a-b68854743ae5","Type":"ContainerStarted","Data":"b0607b75e9752af30c5403f8d635e35589e6411ddd9fffd786a05cdc12de3000"} Oct 30 21:48:55.170354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.170300 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podStartSLOduration=2.170287136 podStartE2EDuration="2.170287136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.167447181 +0000 UTC m=+559.117402318" watchObservedRunningTime="2026-10-30 21:48:55.170287136 +0000 UTC m=+559.120242079" Oct 30 21:48:55.300952 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.300907 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:48:55.302442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.302422 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:48:55.302502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:55.302467 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:48:56.302154 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:56.302114 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:48:56.302660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:56.302181 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:48:57.301858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:57.301820 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:48:57.302032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:57.301875 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:48:58.301169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:58.301134 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:48:58.301644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:58.301187 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:48:59.301634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:59.301601 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:48:59.302013 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:48:59.301653 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:49:00.301922 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:00.301879 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:49:00.302340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:00.301957 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:49:01.302111 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:01.302077 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:49:01.302525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:01.302143 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 21:49:02.735703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:02.735664 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:49:02.736166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:02.736019 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:49:02.736842 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:02.736817 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" Oct 30 21:49:08.999994 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:08.999959 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 21:49:09.008309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.008282 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.022672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.022601 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 21:49:09.066906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.066875 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.066906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.066908 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhlz\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.066941 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067060 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067208 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067275 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067323 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.067370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.067359 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168528 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168605 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhlz\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168681 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168702 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.168764 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168732 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.168813 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169197 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.169175 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.169224 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.169297 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.169355 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.169403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.169369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.171224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.171200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.171494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.171475 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.181790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.181769 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.181870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.181849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhlz\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-v8kmv\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.325778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.325699 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:09.622345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.622321 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 21:49:09.624556 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:09.624528 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4208bd2e_e746_4e58_afc9_058846b00e29.slice/crio-f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd WatchSource:0}: Error finding container f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd: Status 404 returned error can't find the container with id f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd Oct 30 21:49:09.626762 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.626729 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:09.626870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.626797 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:09.626870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:09.626828 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:10.240758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.240719 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" event={"ID":"4208bd2e-e746-4e58-afc9-058846b00e29","Type":"ContainerStarted","Data":"a1a20b3e9f2cd6ee0cae2229b875ccc4ee8822d934e111f8b5288f6fcb0f7896"} Oct 30 21:49:10.241204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.240778 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" event={"ID":"4208bd2e-e746-4e58-afc9-058846b00e29","Type":"ContainerStarted","Data":"f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd"} Oct 30 21:49:10.263487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.263436 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podStartSLOduration=2.263421671 podStartE2EDuration="2.263421671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.26175104 +0000 UTC m=+574.211705987" watchObservedRunningTime="2026-10-30 21:49:10.263421671 +0000 UTC m=+574.213376618" Oct 30 21:49:10.331002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.330951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:10.331426 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.331391 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:10.331630 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:10.331606 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:11.332290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:11.331392 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:11.332290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:11.331459 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:12.331293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:12.326655 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:12.331293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:12.326722 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:13.331668 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:13.331625 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:13.332327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:13.332257 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:14.328761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:14.328720 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:14.329099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:14.329069 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:15.332291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:15.331690 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:15.332291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:15.331771 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:16.326340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:16.326237 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:49:16.326340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:16.326304 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 21:49:17.889134 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:17.889093 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:17.889892 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:17.889855 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:17.891094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:17.891068 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 21:49:18.101902 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:49:18.101844 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:49:29.349620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.349577 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:49:29.353635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.353613 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.368359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.368335 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-367ad753-openshift-default-dockercfg-trg4f\"" Oct 30 21:49:29.378296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.378254 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:49:29.392547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392482 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cmxl\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.392547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392532 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.392741 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392637 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.392741 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392700 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.393407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392790 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.393407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392831 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.393407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.393407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.393407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.392975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.471372 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.471337 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:49:29.475186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.475161 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.478943 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.478920 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-681bcb24-openshift-default-dockercfg-pzw9b\"" Oct 30 21:49:29.493459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493435 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.493585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493467 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.493585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.493585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.493754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493587 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.493754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493616 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493929 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cmxl\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.493995 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494043 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494090 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494232 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494387 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.494741 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.494691 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.496759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.496734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.496895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.496877 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.502215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.502188 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:49:29.519543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.519521 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.530479 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.530451 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cmxl\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\") pod \"e2e-ait-367ad753-openshift-default-95cff747c-zqgx7\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.594864 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.594833 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.594864 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.594868 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.594894 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.594951 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkl4t\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.595040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.595061 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.595088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.595104 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.595377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.595228 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.667997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.667966 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:29.696552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696709 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696618 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696709 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696645 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696709 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696680 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkl4t\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696807 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.696875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696852 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696967 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.696998 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.697206 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.697472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.697572 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.697558 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.699246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.699221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.699692 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.699671 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.712163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.712138 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.712383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.712360 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkl4t\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\") pod \"e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.789449 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.789411 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:29.979222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.979194 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:49:29.981629 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:29.981597 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafba3bb9_1211_4f56_890b_d82a6fc42d43.slice/crio-94b702636a40259b55c7e2b56687290a58838864a99ab20bd3ff082c8464c54d WatchSource:0}: Error finding container 94b702636a40259b55c7e2b56687290a58838864a99ab20bd3ff082c8464c54d: Status 404 returned error can't find the container with id 94b702636a40259b55c7e2b56687290a58838864a99ab20bd3ff082c8464c54d Oct 30 21:49:29.984031 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.983996 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:29.984131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.984076 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:29.984131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:29.984124 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:30.080687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.080657 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:49:30.083221 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:30.083192 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd549f0a4_3a6d_4e52_9037_0b12bac14b30.slice/crio-141bbb0a6271ae7ed3d6e42a33fb6ebf5f5a75fb4252942ed777bcdfcaff74f0 WatchSource:0}: Error finding container 141bbb0a6271ae7ed3d6e42a33fb6ebf5f5a75fb4252942ed777bcdfcaff74f0: Status 404 returned error can't find the container with id 141bbb0a6271ae7ed3d6e42a33fb6ebf5f5a75fb4252942ed777bcdfcaff74f0 Oct 30 21:49:30.086226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.086192 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:30.086335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.086258 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:30.086335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.086314 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:30.352727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.352628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerStarted","Data":"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98"} Oct 30 21:49:30.352727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.352698 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerStarted","Data":"141bbb0a6271ae7ed3d6e42a33fb6ebf5f5a75fb4252942ed777bcdfcaff74f0"} Oct 30 21:49:30.354824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.354791 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerStarted","Data":"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9"} Oct 30 21:49:30.354936 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.354854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerStarted","Data":"94b702636a40259b55c7e2b56687290a58838864a99ab20bd3ff082c8464c54d"} Oct 30 21:49:30.376981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.376918 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podStartSLOduration=1.376901132 podStartE2EDuration="1.376901132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.37634689 +0000 UTC m=+594.326301872" watchObservedRunningTime="2026-10-30 21:49:30.376901132 +0000 UTC m=+594.326856079" Oct 30 21:49:30.406253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.406182 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podStartSLOduration=1.406162197 podStartE2EDuration="1.406162197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.401069978 +0000 UTC m=+594.351024947" watchObservedRunningTime="2026-10-30 21:49:30.406162197 +0000 UTC m=+594.356117144" Oct 30 21:49:30.668612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.668576 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:30.790166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:30.790133 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:31.244231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.244189 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:49:31.248672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.248648 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.266008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.265981 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:49:31.289574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.289546 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9528ebfc-openshift-default-dockercfg-9g9nx\"" Oct 30 21:49:31.315192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315160 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h28\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315212 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315330 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315372 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315476 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.315661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.315651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416617 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416721 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416781 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416848 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416884 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.416930 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.417066 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h28\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.417144 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.420292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.419914 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.421360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.420926 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.421360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.421348 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.424721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.421566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.424721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.422067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.427875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.427825 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.434041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.434009 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.445405 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.445249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h28\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.464182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.464150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\") pod \"e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.568033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.567949 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:31.675040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.675006 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:31.795820 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.795795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:31.868984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.868956 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:49:31.871370 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:31.871344 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621c551d_484c_46ca_bc34_d3d678a1538e.slice/crio-8d14efd6866943fa416fbdc05d84d137a27c12b2bd2dd20a94da34dcd2798460 WatchSource:0}: Error finding container 8d14efd6866943fa416fbdc05d84d137a27c12b2bd2dd20a94da34dcd2798460: Status 404 returned error can't find the container with id 8d14efd6866943fa416fbdc05d84d137a27c12b2bd2dd20a94da34dcd2798460 Oct 30 21:49:31.874301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.874250 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:31.874390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.874337 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:31.874390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:31.874366 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:32.368538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.368494 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerStarted","Data":"5b1a5f46fb1c93ebdfd9dfa458245382abc6c3114d0dad166d48ec7d5a9143cd"} Oct 30 21:49:32.368749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.368565 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerStarted","Data":"8d14efd6866943fa416fbdc05d84d137a27c12b2bd2dd20a94da34dcd2798460"} Oct 30 21:49:32.368749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.368597 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:32.369107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.369076 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:32.369858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.369835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:49:32.369976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.369960 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:32.393902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.393835 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podStartSLOduration=1.393817197 podStartE2EDuration="1.393817197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:32.393018275 +0000 UTC m=+596.342973227" watchObservedRunningTime="2026-10-30 21:49:32.393817197 +0000 UTC m=+596.343772146" Oct 30 21:49:32.576315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.569726 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:32.578249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:32.578221 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:33.254989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.254960 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:49:33.259221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.259192 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.264416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.264217 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-31a18d9b-dockercfg-6bj7n\"" Oct 30 21:49:33.264416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.264223 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-31a18d9b\"" Oct 30 21:49:33.269415 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.269012 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:49:33.301162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.301124 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:49:33.306730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.306700 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.315897 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.315874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:49:33.338625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.338593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pls98\" (UniqueName: \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.338765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.338671 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.386195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.386161 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:33.390289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.387247 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:33.439597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.439552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxwl\" (UniqueName: \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.439783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.439658 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.439783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.439715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.439999 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.439972 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pls98\" (UniqueName: \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.441644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.441615 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.452209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.452179 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pls98\" (UniqueName: \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\") pod \"payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.541439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.541353 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxwl\" (UniqueName: \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.541439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.541417 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.542161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.542141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.554650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.554621 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxwl\" (UniqueName: \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\") pod \"payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.576531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.576507 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:33.624146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.624019 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:33.870829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.870798 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:49:33.872251 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:33.872224 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38fb764_292a_4ce3_b1bc_cb3ce2456498.slice/crio-caa48bb432205823c33772e8bfc68aea323760d9ae27f2ec9ff1febb13db1604 WatchSource:0}: Error finding container caa48bb432205823c33772e8bfc68aea323760d9ae27f2ec9ff1febb13db1604: Status 404 returned error can't find the container with id caa48bb432205823c33772e8bfc68aea323760d9ae27f2ec9ff1febb13db1604 Oct 30 21:49:33.936242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:33.936215 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:49:33.942007 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:33.941982 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de95922_eed9_452e_a6dd_f02364b88916.slice/crio-0b5dfe667a16c7f223d3c0450297c951c0155a9c19a94203e4e59865285a1d53 WatchSource:0}: Error finding container 0b5dfe667a16c7f223d3c0450297c951c0155a9c19a94203e4e59865285a1d53: Status 404 returned error can't find the container with id 0b5dfe667a16c7f223d3c0450297c951c0155a9c19a94203e4e59865285a1d53 Oct 30 21:49:34.018740 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.018714 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:49:34.022530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.022510 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.026490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.026449 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-367ad753-dockercfg-xv9cj\"" Oct 30 21:49:34.037433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.037412 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:49:34.081740 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.081710 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:49:34.085959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.085942 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.091639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.091611 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-367ad753\"" Oct 30 21:49:34.093964 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.093933 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:49:34.168823 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.157929 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrl4\" (UniqueName: \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.170816 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.170786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rktp\" (UniqueName: \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.171121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.171102 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.171360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.171338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.272826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.272794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrl4\" (UniqueName: \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.272826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.272838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rktp\" (UniqueName: \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.273048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.272887 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.273048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.272954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.273657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.273639 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.273717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.273698 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.323427 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.323395 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rktp\" (UniqueName: \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\") pod \"payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.325421 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.325387 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrl4\" (UniqueName: \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\") pod \"payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.336346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.336323 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:34.396127 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.396053 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" event={"ID":"f38fb764-292a-4ce3-b1bc-cb3ce2456498","Type":"ContainerStarted","Data":"533817533fad7b3d4c649dbf9222644ca6d3b25421c3435c4cdd670741e5a9cc"} Oct 30 21:49:34.396433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.396359 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:34.396433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.396409 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" event={"ID":"f38fb764-292a-4ce3-b1bc-cb3ce2456498","Type":"ContainerStarted","Data":"caa48bb432205823c33772e8bfc68aea323760d9ae27f2ec9ff1febb13db1604"} Oct 30 21:49:34.398927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.398897 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" event={"ID":"4de95922-eed9-452e-a6dd-f02364b88916","Type":"ContainerStarted","Data":"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552"} Oct 30 21:49:34.399059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.398955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" event={"ID":"4de95922-eed9-452e-a6dd-f02364b88916","Type":"ContainerStarted","Data":"0b5dfe667a16c7f223d3c0450297c951c0155a9c19a94203e4e59865285a1d53"} Oct 30 21:49:34.405563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.405489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:34.490790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.490679 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" podStartSLOduration=1.490665052 podStartE2EDuration="1.490665052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:34.469157201 +0000 UTC m=+598.419112149" watchObservedRunningTime="2026-10-30 21:49:34.490665052 +0000 UTC m=+598.440620026" Oct 30 21:49:34.494468 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.493825 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" podStartSLOduration=1.4937695739999999 podStartE2EDuration="1.493769574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:34.44752733 +0000 UTC m=+598.397482279" watchObservedRunningTime="2026-10-30 21:49:34.493769574 +0000 UTC m=+598.443724521" Oct 30 21:49:34.638329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.638303 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:49:34.640627 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:34.640593 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12c6010_be0d_4f4a_a872_c09001c44544.slice/crio-6021908726c1b70214eafed098047bd72223984f4ad1f093145ed4ab7f20856c WatchSource:0}: Error finding container 6021908726c1b70214eafed098047bd72223984f4ad1f093145ed4ab7f20856c: Status 404 returned error can't find the container with id 6021908726c1b70214eafed098047bd72223984f4ad1f093145ed4ab7f20856c Oct 30 21:49:34.733602 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:34.733574 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:49:34.736147 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:34.736114 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bebf2d_37b2_4892_b5c6_c79605e672f6.slice/crio-14249677f88e93a5a3c4ef060676fca4abdffedf5ff7191fe251a25592388b1f WatchSource:0}: Error finding container 14249677f88e93a5a3c4ef060676fca4abdffedf5ff7191fe251a25592388b1f: Status 404 returned error can't find the container with id 14249677f88e93a5a3c4ef060676fca4abdffedf5ff7191fe251a25592388b1f Oct 30 21:49:35.405724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.405689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" event={"ID":"a12c6010-be0d-4f4a-a872-c09001c44544","Type":"ContainerStarted","Data":"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7"} Oct 30 21:49:35.405909 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.405749 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" event={"ID":"a12c6010-be0d-4f4a-a872-c09001c44544","Type":"ContainerStarted","Data":"6021908726c1b70214eafed098047bd72223984f4ad1f093145ed4ab7f20856c"} Oct 30 21:49:35.405909 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.405889 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:35.407377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.407352 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" event={"ID":"94bebf2d-37b2-4892-b5c6-c79605e672f6","Type":"ContainerStarted","Data":"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080"} Oct 30 21:49:35.407519 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.407396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" event={"ID":"94bebf2d-37b2-4892-b5c6-c79605e672f6","Type":"ContainerStarted","Data":"14249677f88e93a5a3c4ef060676fca4abdffedf5ff7191fe251a25592388b1f"} Oct 30 21:49:35.407578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.407548 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:35.407632 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.407585 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:35.445655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.445602 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" podStartSLOduration=2.445589219 podStartE2EDuration="2.445589219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.441869447 +0000 UTC m=+599.391824394" watchObservedRunningTime="2026-10-30 21:49:35.445589219 +0000 UTC m=+599.395544165" Oct 30 21:49:35.869495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.869392 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" podStartSLOduration=1.869375937 podStartE2EDuration="1.869375937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.486541592 +0000 UTC m=+599.436496541" watchObservedRunningTime="2026-10-30 21:49:35.869375937 +0000 UTC m=+599.819330882" Oct 30 21:49:35.869877 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.869859 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:49:35.873898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.873880 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:35.893252 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.893221 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9528ebfc-dockercfg-4gnr7\"" Oct 30 21:49:35.893846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.893825 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9528ebfc\"" Oct 30 21:49:35.896935 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.896889 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:49:35.901179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.901159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:49:35.901325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.901255 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:35.935624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:35.935595 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:49:36.029307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.021770 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.029307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.021827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.029307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.021985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292f2\" (UniqueName: \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.029307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.022011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.123311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.123185 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.123488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.123364 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292f2\" (UniqueName: \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.123488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.123392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.123488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.123467 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.124154 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.124130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.124454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.124431 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.141886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.141861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\") pod \"payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.141995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.141951 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292f2\" (UniqueName: \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\") pod \"payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.190403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.190370 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9528ebfc-dockercfg-4gnr7\"" Oct 30 21:49:36.198788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.198764 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:36.217543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.217521 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:36.303492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.301737 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:49:36.377829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.377742 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:49:36.500088 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.500065 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:49:36.502298 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:36.502258 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aba5615_53f7_4f45_9291_2b492f558a82.slice/crio-56fc8e795075e5a4d47e8d8cd1b421d008b353f3aac2923deaed4c04eef144c7 WatchSource:0}: Error finding container 56fc8e795075e5a4d47e8d8cd1b421d008b353f3aac2923deaed4c04eef144c7: Status 404 returned error can't find the container with id 56fc8e795075e5a4d47e8d8cd1b421d008b353f3aac2923deaed4c04eef144c7 Oct 30 21:49:36.553786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:36.553700 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:49:36.556009 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:36.555981 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20632c7_f976_4027_b9ba_3478a2ca6b3e.slice/crio-b1cd12fba6a321f45b88e768a1e2de629ece8c553160d4936f82408c6cc1f54b WatchSource:0}: Error finding container b1cd12fba6a321f45b88e768a1e2de629ece8c553160d4936f82408c6cc1f54b: Status 404 returned error can't find the container with id b1cd12fba6a321f45b88e768a1e2de629ece8c553160d4936f82408c6cc1f54b Oct 30 21:49:37.421240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.421204 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" event={"ID":"7aba5615-53f7-4f45-9291-2b492f558a82","Type":"ContainerStarted","Data":"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893"} Oct 30 21:49:37.421698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.421260 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" event={"ID":"7aba5615-53f7-4f45-9291-2b492f558a82","Type":"ContainerStarted","Data":"56fc8e795075e5a4d47e8d8cd1b421d008b353f3aac2923deaed4c04eef144c7"} Oct 30 21:49:37.421698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.421413 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:37.423207 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.423173 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" event={"ID":"e20632c7-f976-4027-b9ba-3478a2ca6b3e","Type":"ContainerStarted","Data":"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af"} Oct 30 21:49:37.423357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.423221 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" event={"ID":"e20632c7-f976-4027-b9ba-3478a2ca6b3e","Type":"ContainerStarted","Data":"b1cd12fba6a321f45b88e768a1e2de629ece8c553160d4936f82408c6cc1f54b"} Oct 30 21:49:37.423357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.423256 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:37.440943 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.440899 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" podStartSLOduration=2.440886814 podStartE2EDuration="2.440886814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.438155893 +0000 UTC m=+601.388110832" watchObservedRunningTime="2026-10-30 21:49:37.440886814 +0000 UTC m=+601.390841760" Oct 30 21:49:37.456794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:37.456748 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" podStartSLOduration=2.456735351 podStartE2EDuration="2.456735351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.454831859 +0000 UTC m=+601.404786805" watchObservedRunningTime="2026-10-30 21:49:37.456735351 +0000 UTC m=+601.406690321" Oct 30 21:49:39.160238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.160202 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:49:39.164991 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.164968 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.172960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.172941 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4487d16a-openshift-default-dockercfg-gk4fx\"" Oct 30 21:49:39.187830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.187805 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:49:39.271051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271015 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklwn\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271190 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271254 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271318 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271430 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271379 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271508 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271492 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.271570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.271557 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.372830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.372792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.372830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.372831 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.372854 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.372928 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373009 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373066 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373115 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklwn\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373146 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373183 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373420 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.373676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373653 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.374104 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373780 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.374104 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.373801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.374104 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.374063 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.375502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.375478 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.375873 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.375852 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.386664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.386628 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklwn\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.386664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.386660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\") pod \"e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.482980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.482897 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:39.784097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.784072 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:49:39.784582 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:39.784551 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5052911_0326_45e8_b13a_6687d3048d88.slice/crio-03047db81c196909a1b562d0ab85ff68a342f456a0a38aa939a23c239c8f3197 WatchSource:0}: Error finding container 03047db81c196909a1b562d0ab85ff68a342f456a0a38aa939a23c239c8f3197: Status 404 returned error can't find the container with id 03047db81c196909a1b562d0ab85ff68a342f456a0a38aa939a23c239c8f3197 Oct 30 21:49:39.787043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.787011 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:39.787120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.787076 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:39.787120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:39.787107 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:40.479147 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:40.479102 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerStarted","Data":"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67"} Oct 30 21:49:40.479147 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:40.479154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerStarted","Data":"03047db81c196909a1b562d0ab85ff68a342f456a0a38aa939a23c239c8f3197"} Oct 30 21:49:40.483122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:40.483099 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:40.488161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:40.488142 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:40.502118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:40.502060 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" podStartSLOduration=1.502047404 podStartE2EDuration="1.502047404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.500685452 +0000 UTC m=+604.450640391" watchObservedRunningTime="2026-10-30 21:49:40.502047404 +0000 UTC m=+604.452002350" Oct 30 21:49:41.485423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:41.485396 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:41.486442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:41.486425 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:49:42.672376 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.672335 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd"] Oct 30 21:49:42.676645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.676626 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.686599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.686080 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd"] Oct 30 21:49:42.687145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.687116 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:42.687253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.687235 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:42.687576 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.687557 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:42.687662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.687649 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:49:42.708799 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.708767 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.709140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.708835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rks2\" (UniqueName: \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.709140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.708865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.810001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.809972 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rks2\" (UniqueName: \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.810184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.810004 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.810184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.810109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.810917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.810889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.824289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.818697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.835518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.832924 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rks2\" (UniqueName: \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\") pod \"maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:42.995389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:42.995304 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:43.390114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.390071 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:49:43.396015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.395988 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:49:43.396254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.396132 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.429302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.421466 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4487d16a\"" Oct 30 21:49:43.482256 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.482219 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd"] Oct 30 21:49:43.492880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.489069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.492880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.492570 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdcl\" (UniqueName: \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.601300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.594938 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.615454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.615420 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdcl\" (UniqueName: \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.629295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.602412 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.674288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.673098 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:49:43.678747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.678722 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdcl\" (UniqueName: \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\") pod \"payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:43.789526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:43.789489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:44.176302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.174968 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:49:44.190292 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:44.179765 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555e9fdf_aa03_4fa3_8137_2b9e3b241ad5.slice/crio-5d486547ed2b49ee6735e96d5cbe602c56a9e75a18264e8938830b17b940c552 WatchSource:0}: Error finding container 5d486547ed2b49ee6735e96d5cbe602c56a9e75a18264e8938830b17b940c552: Status 404 returned error can't find the container with id 5d486547ed2b49ee6735e96d5cbe602c56a9e75a18264e8938830b17b940c552 Oct 30 21:49:44.674617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.674571 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" event={"ID":"75b19a45-e639-4922-8cb7-a026fcd48f81","Type":"ContainerStarted","Data":"abf2f8175569adce784e08b13739376acffca93d6f11d8ce24a5035c52ebec56"} Oct 30 21:49:44.676460 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.676430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" event={"ID":"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5","Type":"ContainerStarted","Data":"41452124dc8032e91478847482d5a1f01c71cdff57d02c5c491817e56773691a"} Oct 30 21:49:44.676566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.676494 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" event={"ID":"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5","Type":"ContainerStarted","Data":"5d486547ed2b49ee6735e96d5cbe602c56a9e75a18264e8938830b17b940c552"} Oct 30 21:49:44.676674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.676637 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:44.694696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:44.694650 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" podStartSLOduration=1.694612725 podStartE2EDuration="1.694612725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:44.693599194 +0000 UTC m=+608.643554141" watchObservedRunningTime="2026-10-30 21:49:44.694612725 +0000 UTC m=+608.644567670" Oct 30 21:49:45.431657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:45.431466 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:49:46.441304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:46.432998 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:49:46.441304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:46.440159 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:49:46.441847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:46.441438 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:49:48.117951 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:49:48.117874 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:49:48.416298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:48.410453 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:49:48.416298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:48.410523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:48.443525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:48.443467 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:49:48.443716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:48.443554 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:49:49.441468 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:49.441416 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:49:49.441981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:49.441508 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:49.785819 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:49.785780 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" event={"ID":"75b19a45-e639-4922-8cb7-a026fcd48f81","Type":"ContainerStarted","Data":"6cfcf88db80db529b37357d51781e9030df8483be768894c41c51a40524b031c"} Oct 30 21:49:49.844447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:49.843972 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" podStartSLOduration=2.616788502 podStartE2EDuration="7.843957347s" totalImagesPullingTime="5.227168845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:49.843816896 +0000 UTC m=+613.793771843" watchObservedRunningTime="2026-10-30 21:49:49.843957347 +0000 UTC m=+613.793912292" Oct 30 21:49:50.796577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:50.796530 5036 generic.go:356] "Generic (PLEG): container finished" podID="75b19a45-e639-4922-8cb7-a026fcd48f81" containerID="6cfcf88db80db529b37357d51781e9030df8483be768894c41c51a40524b031c" exitCode=0 Oct 30 21:49:50.797221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:50.796617 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" event={"ID":"75b19a45-e639-4922-8cb7-a026fcd48f81","Type":"ContainerDied","Data":"6cfcf88db80db529b37357d51781e9030df8483be768894c41c51a40524b031c"} Oct 30 21:49:51.805713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.805664 5036 generic.go:356] "Generic (PLEG): container finished" podID="621c551d-484c-46ca-bc34-d3d678a1538e" containerID="5b1a5f46fb1c93ebdfd9dfa458245382abc6c3114d0dad166d48ec7d5a9143cd" exitCode=0 Oct 30 21:49:51.806333 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.805752 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerDied","Data":"5b1a5f46fb1c93ebdfd9dfa458245382abc6c3114d0dad166d48ec7d5a9143cd"} Oct 30 21:49:51.806333 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.806254 5036 scope.go:122] "RemoveContainer" containerID="5b1a5f46fb1c93ebdfd9dfa458245382abc6c3114d0dad166d48ec7d5a9143cd" Oct 30 21:49:51.808124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808100 5036 generic.go:356] "Generic (PLEG): container finished" podID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerID="ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9" exitCode=0 Oct 30 21:49:51.808292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808122 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerDied","Data":"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9"} Oct 30 21:49:51.808292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808092 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:51.808715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808341 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:51.808715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808410 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:51.810396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.808937 5036 scope.go:122] "RemoveContainer" containerID="ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9" Oct 30 21:49:51.810992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.810960 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:51.811112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.811040 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:51.811112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:51.811084 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:52.054908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.054881 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:52.172756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.172722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\") pod \"75b19a45-e639-4922-8cb7-a026fcd48f81\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " Oct 30 21:49:52.173014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.172995 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\") pod \"75b19a45-e639-4922-8cb7-a026fcd48f81\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " Oct 30 21:49:52.173194 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.173177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\" (UniqueName: \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\") pod \"75b19a45-e639-4922-8cb7-a026fcd48f81\" (UID: \"75b19a45-e639-4922-8cb7-a026fcd48f81\") " Oct 30 21:49:52.173671 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.173643 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca" pod "75b19a45-e639-4922-8cb7-a026fcd48f81" (UID: "75b19a45-e639-4922-8cb7-a026fcd48f81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:52.173968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.173949 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:52.177453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.177414 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2" pod "75b19a45-e639-4922-8cb7-a026fcd48f81" (UID: "75b19a45-e639-4922-8cb7-a026fcd48f81"). InnerVolumeSpecName "kube-api-access-7rks2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:52.177707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.177688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca" pod "75b19a45-e639-4922-8cb7-a026fcd48f81" (UID: "75b19a45-e639-4922-8cb7-a026fcd48f81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:52.275760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.275378 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b19a45-e639-4922-8cb7-a026fcd48f81-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:52.275760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.275418 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rks2\" (UniqueName: \"kubernetes.io/projected/75b19a45-e639-4922-8cb7-a026fcd48f81-kube-api-access-7rks2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:52.879915 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.879880 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerStarted","Data":"fdaa8ba7513c53d8e6ef447a8dc6e545dde9fb28cb839b8a25d6ee1bbf5307ec"} Oct 30 21:49:52.882011 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.881993 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" Oct 30 21:49:52.882223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.882023 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd" event={"ID":"75b19a45-e639-4922-8cb7-a026fcd48f81","Type":"ContainerDied","Data":"abf2f8175569adce784e08b13739376acffca93d6f11d8ce24a5035c52ebec56"} Oct 30 21:49:52.882440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.882423 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf2f8175569adce784e08b13739376acffca93d6f11d8ce24a5035c52ebec56" Oct 30 21:49:52.885236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:52.885188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerStarted","Data":"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa"} Oct 30 21:49:53.077208 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.077169 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64"] Oct 30 21:49:53.078045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.078020 5036 cpu_manager.go:414] "containerMap: removing container" podUID="75b19a45-e639-4922-8cb7-a026fcd48f81" containerName="revoke-keys" Oct 30 21:49:53.078045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.078044 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b19a45-e639-4922-8cb7-a026fcd48f81" containerName="revoke-keys" Oct 30 21:49:53.078227 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.078167 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b19a45-e639-4922-8cb7-a026fcd48f81" containerName="revoke-keys" Oct 30 21:49:53.087302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.084149 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.088101 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.087449 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64"] Oct 30 21:49:53.088849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.088816 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:53.089150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.089042 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:53.089394 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.089102 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:53.089504 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.089343 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:49:53.188878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.188840 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.189051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.188928 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4jt\" (UniqueName: \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.189051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.188958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.289976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.289937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.290138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.290110 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4jt\" (UniqueName: \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.290222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.290174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.291192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.291168 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.292844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.292823 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.302918 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.302900 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4jt\" (UniqueName: \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\") pod \"maas-api-revoke-keys-e2e-ait-367ad753-9ph64\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.403042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.403006 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7"] Oct 30 21:49:53.407531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.407509 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:53.407673 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.407652 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.451413 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.451342 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7"] Oct 30 21:49:53.499001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.498953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.499196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.499063 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.499255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.499191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrr6\" (UniqueName: \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.577289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.571512 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:53.577289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.572016 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:49:53.577289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.572080 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:49:53.600363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.600337 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.601295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.600784 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrr6\" (UniqueName: \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.601295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.600876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.601962 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.601910 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.603293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.603253 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.669063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.668637 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:53.669063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.668764 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:49:53.669063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.668806 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 21:49:53.681600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.681379 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrr6\" (UniqueName: \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\") pod \"maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.755293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.754560 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:53.983807 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:53.981356 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64"] Oct 30 21:49:53.985332 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:49:53.985306 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09974e4c_dbf6_46c5_b079_e50706be3871.slice/crio-7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2 WatchSource:0}: Error finding container 7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2: Status 404 returned error can't find the container with id 7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2 Oct 30 21:49:54.179218 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.179158 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7"] Oct 30 21:49:54.569119 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.569082 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:49:54.569450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.569419 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:49:54.671146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.671003 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:49:54.671146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.671080 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 21:49:54.904789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.904746 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" event={"ID":"f8a890b0-9161-42c3-aeec-27918c58adfc","Type":"ContainerStarted","Data":"b11d98e7de498de7eecc97ef7eeec1382225c42b7a81b3a143bd44a50b133234"} Oct 30 21:49:54.904961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.904809 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" event={"ID":"f8a890b0-9161-42c3-aeec-27918c58adfc","Type":"ContainerStarted","Data":"19496420fb24649f3226b0de4f0b1c2650ce0fd39f51e367c73b47a0b81a3db6"} Oct 30 21:49:54.906753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.906716 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" event={"ID":"09974e4c-dbf6-46c5-b079-e50706be3871","Type":"ContainerStarted","Data":"dd348b5921a48bf7aaf3ab137b0a36d88bb16507837fe28fd7076e4c8ace7a72"} Oct 30 21:49:54.906875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.906764 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" event={"ID":"09974e4c-dbf6-46c5-b079-e50706be3871","Type":"ContainerStarted","Data":"7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2"} Oct 30 21:49:54.934301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:54.928778 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" podStartSLOduration=1.928758689 podStartE2EDuration="1.928758689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:54.926556412 +0000 UTC m=+618.876511360" watchObservedRunningTime="2026-10-30 21:49:54.928758689 +0000 UTC m=+618.878713637" Oct 30 21:49:55.569373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.568939 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:49:55.569373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.568986 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:49:55.671300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.670679 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:49:55.671300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.670742 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 21:49:55.768536 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.768492 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:49:55.795917 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.795844 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" podStartSLOduration=2.795821359 podStartE2EDuration="2.795821359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:54.966558437 +0000 UTC m=+618.916513384" watchObservedRunningTime="2026-10-30 21:49:55.795821359 +0000 UTC m=+619.745776322" Oct 30 21:49:55.979538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.979492 5036 generic.go:356] "Generic (PLEG): container finished" podID="f8a890b0-9161-42c3-aeec-27918c58adfc" containerID="b11d98e7de498de7eecc97ef7eeec1382225c42b7a81b3a143bd44a50b133234" exitCode=0 Oct 30 21:49:55.979727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.979567 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" event={"ID":"f8a890b0-9161-42c3-aeec-27918c58adfc","Type":"ContainerDied","Data":"b11d98e7de498de7eecc97ef7eeec1382225c42b7a81b3a143bd44a50b133234"} Oct 30 21:49:55.981564 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.981537 5036 generic.go:356] "Generic (PLEG): container finished" podID="09974e4c-dbf6-46c5-b079-e50706be3871" containerID="dd348b5921a48bf7aaf3ab137b0a36d88bb16507837fe28fd7076e4c8ace7a72" exitCode=0 Oct 30 21:49:55.981704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:55.981651 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" event={"ID":"09974e4c-dbf6-46c5-b079-e50706be3871","Type":"ContainerDied","Data":"dd348b5921a48bf7aaf3ab137b0a36d88bb16507837fe28fd7076e4c8ace7a72"} Oct 30 21:49:56.575502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:56.575438 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:49:56.576037 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:56.575522 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:49:56.673290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:56.668682 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:49:56.673290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:56.668744 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 21:49:57.622638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:57.603568 5036 patch_prober.go:29] interesting pod/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:49:57.622638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:57.603637 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:49:57.799291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:57.763591 5036 patch_prober.go:29] interesting pod/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:49:57.799291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:57.763650 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 21:49:58.165355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.165236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" event={"ID":"f8a890b0-9161-42c3-aeec-27918c58adfc","Type":"ContainerDied","Data":"19496420fb24649f3226b0de4f0b1c2650ce0fd39f51e367c73b47a0b81a3db6"} Oct 30 21:49:58.165627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.165587 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19496420fb24649f3226b0de4f0b1c2650ce0fd39f51e367c73b47a0b81a3db6" Oct 30 21:49:58.195678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.194682 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:49:58.201852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.201825 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.320680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\") pod \"f8a890b0-9161-42c3-aeec-27918c58adfc\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.320833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\" (UniqueName: \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\") pod \"f8a890b0-9161-42c3-aeec-27918c58adfc\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.320874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\") pod \"09974e4c-dbf6-46c5-b079-e50706be3871\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.320919 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\") pod \"09974e4c-dbf6-46c5-b079-e50706be3871\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.320984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\" (UniqueName: \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\") pod \"09974e4c-dbf6-46c5-b079-e50706be3871\" (UID: \"09974e4c-dbf6-46c5-b079-e50706be3871\") " Oct 30 21:49:58.334292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.321046 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\") pod \"f8a890b0-9161-42c3-aeec-27918c58adfc\" (UID: \"f8a890b0-9161-42c3-aeec-27918c58adfc\") " Oct 30 21:49:58.350288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.346770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca" pod "f8a890b0-9161-42c3-aeec-27918c58adfc" (UID: "f8a890b0-9161-42c3-aeec-27918c58adfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:58.350947 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.350647 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca" pod "09974e4c-dbf6-46c5-b079-e50706be3871" (UID: "09974e4c-dbf6-46c5-b079-e50706be3871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:58.351868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.351829 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca" pod "f8a890b0-9161-42c3-aeec-27918c58adfc" (UID: "f8a890b0-9161-42c3-aeec-27918c58adfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:58.354721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.354666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6" pod "f8a890b0-9161-42c3-aeec-27918c58adfc" (UID: "f8a890b0-9161-42c3-aeec-27918c58adfc"). InnerVolumeSpecName "kube-api-access-hkrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:58.354856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.354791 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt" pod "09974e4c-dbf6-46c5-b079-e50706be3871" (UID: "09974e4c-dbf6-46c5-b079-e50706be3871"). InnerVolumeSpecName "kube-api-access-6r4jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:58.356464 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.356437 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca" pod "09974e4c-dbf6-46c5-b079-e50706be3871" (UID: "09974e4c-dbf6-46c5-b079-e50706be3871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423042 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423086 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a890b0-9161-42c3-aeec-27918c58adfc-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423103 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrr6\" (UniqueName: \"kubernetes.io/projected/f8a890b0-9161-42c3-aeec-27918c58adfc-kube-api-access-hkrr6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423123 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423141 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09974e4c-dbf6-46c5-b079-e50706be3871-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.423157 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4jt\" (UniqueName: \"kubernetes.io/projected/09974e4c-dbf6-46c5-b079-e50706be3871-kube-api-access-6r4jt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.572757 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.572725 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:58.573122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.573052 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:58.573844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.573822 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:49:58.673814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.673743 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:58.674219 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.674113 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:58.675155 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:58.675130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:49:59.182031 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.182001 5036 generic.go:356] "Generic (PLEG): container finished" podID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerID="843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98" exitCode=0 Oct 30 21:49:59.182202 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.182078 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerDied","Data":"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98"} Oct 30 21:49:59.182451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.182429 5036 scope.go:122] "RemoveContainer" containerID="843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98" Oct 30 21:49:59.183666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183646 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" Oct 30 21:49:59.183793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183664 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:59.183793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183730 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:59.183793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64" event={"ID":"09974e4c-dbf6-46c5-b079-e50706be3871","Type":"ContainerDied","Data":"7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2"} Oct 30 21:49:59.183793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183766 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:59.183793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183777 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec269076227d3a3418e9c4818c9dbcf4dea96e33b0da93d99becb1e8ada40a2" Oct 30 21:49:59.184045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:49:59.183865 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7" Oct 30 21:50:00.163553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.163515 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:50:00.175496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.175457 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:50:00.198712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.198685 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerStarted","Data":"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d"} Oct 30 21:50:00.790433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.790402 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:00.790629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.790560 5036 patch_prober.go:29] interesting pod/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:50:00.790629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:00.790608 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 21:50:01.790806 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:01.790763 5036 patch_prober.go:29] interesting pod/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.791195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:01.790837 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 21:50:02.369651 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:02.369611 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:02.790377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:02.790338 5036 patch_prober.go:29] interesting pod/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.790570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:02.790401 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 21:50:02.850665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:02.850630 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:50:02.860343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:02.860309 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:50:03.204226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:03.204192 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:50:03.223690 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:03.216987 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:50:03.790621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:03.790581 5036 patch_prober.go:29] interesting pod/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:50:03.790822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:03.790652 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 21:50:04.789928 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:04.789892 5036 patch_prober.go:29] interesting pod/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:50:04.790339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:04.789951 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 21:50:05.046080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.045252 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:50:05.046080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.045499 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" containerID="cri-o://fdaa8ba7513c53d8e6ef447a8dc6e545dde9fb28cb839b8a25d6ee1bbf5307ec" gracePeriod=30 Oct 30 21:50:05.164781 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.164723 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" podUID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" containerName="payload-processing" containerID="cri-o://915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af" gracePeriod=25 Oct 30 21:50:05.175812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.175752 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" podUID="7aba5615-53f7-4f45-9291-2b492f558a82" containerName="payload-pre-processing" containerID="cri-o://869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893" gracePeriod=25 Oct 30 21:50:05.254008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.253973 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd"] Oct 30 21:50:05.257523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.257406 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-31a18d9b-kjtcd"] Oct 30 21:50:05.499121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.497329 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64"] Oct 30 21:50:05.513381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.513311 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-367ad753-9ph64"] Oct 30 21:50:05.537830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.537799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:50:05.579287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.577985 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:50:05.647308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.615648 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\") pod \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " Oct 30 21:50:05.647308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.615706 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\" (UniqueName: \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\") pod \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\" (UID: \"e20632c7-f976-4027-b9ba-3478a2ca6b3e\") " Oct 30 21:50:05.647308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.634460 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2" pod "e20632c7-f976-4027-b9ba-3478a2ca6b3e" (UID: "e20632c7-f976-4027-b9ba-3478a2ca6b3e"). InnerVolumeSpecName "kube-api-access-292f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:05.647308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.640491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume" pod "e20632c7-f976-4027-b9ba-3478a2ca6b3e" (UID: "e20632c7-f976-4027-b9ba-3478a2ca6b3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:05.684219 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.684181 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:50:05.684500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.684473 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" containerID="cri-o://1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa" gracePeriod=30 Oct 30 21:50:05.713612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.708568 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7"] Oct 30 21:50:05.713612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.711672 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681bcb24-vfgs7"] Oct 30 21:50:05.716870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.716722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\") pod \"7aba5615-53f7-4f45-9291-2b492f558a82\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " Oct 30 21:50:05.717025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.716913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\") pod \"7aba5615-53f7-4f45-9291-2b492f558a82\" (UID: \"7aba5615-53f7-4f45-9291-2b492f558a82\") " Oct 30 21:50:05.717317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.717297 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20632c7-f976-4027-b9ba-3478a2ca6b3e-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.717375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.717324 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292f2\" (UniqueName: \"kubernetes.io/projected/e20632c7-f976-4027-b9ba-3478a2ca6b3e-kube-api-access-292f2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.717929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.717907 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume" pod "7aba5615-53f7-4f45-9291-2b492f558a82" (UID: "7aba5615-53f7-4f45-9291-2b492f558a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:05.720429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.720396 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v" pod "7aba5615-53f7-4f45-9291-2b492f558a82" (UID: "7aba5615-53f7-4f45-9291-2b492f558a82"). InnerVolumeSpecName "kube-api-access-rnh4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:05.818707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.818629 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aba5615-53f7-4f45-9291-2b492f558a82-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.818707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.818667 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/7aba5615-53f7-4f45-9291-2b492f558a82-kube-api-access-rnh4v\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.930313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.929795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:05.931235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:05.931207 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:06.188163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.188131 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09974e4c-dbf6-46c5-b079-e50706be3871" path="/var/lib/kubelet/pods/09974e4c-dbf6-46c5-b079-e50706be3871/volumes" Oct 30 21:50:06.188602 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.188587 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b19a45-e639-4922-8cb7-a026fcd48f81" path="/var/lib/kubelet/pods/75b19a45-e639-4922-8cb7-a026fcd48f81/volumes" Oct 30 21:50:06.188940 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.188926 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a890b0-9161-42c3-aeec-27918c58adfc" path="/var/lib/kubelet/pods/f8a890b0-9161-42c3-aeec-27918c58adfc/volumes" Oct 30 21:50:06.299275 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.299239 5036 generic.go:356] "Generic (PLEG): container finished" podID="7aba5615-53f7-4f45-9291-2b492f558a82" containerID="869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893" exitCode=0 Oct 30 21:50:06.299428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.299341 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" Oct 30 21:50:06.299428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.299334 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" event={"ID":"7aba5615-53f7-4f45-9291-2b492f558a82","Type":"ContainerDied","Data":"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893"} Oct 30 21:50:06.299561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.299476 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb" event={"ID":"7aba5615-53f7-4f45-9291-2b492f558a82","Type":"ContainerDied","Data":"56fc8e795075e5a4d47e8d8cd1b421d008b353f3aac2923deaed4c04eef144c7"} Oct 30 21:50:06.299561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.299509 5036 scope.go:122] "RemoveContainer" containerID="869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893" Oct 30 21:50:06.301130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.301110 5036 generic.go:356] "Generic (PLEG): container finished" podID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" containerID="915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af" exitCode=0 Oct 30 21:50:06.301327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.301309 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" Oct 30 21:50:06.301437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.301358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" event={"ID":"e20632c7-f976-4027-b9ba-3478a2ca6b3e","Type":"ContainerDied","Data":"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af"} Oct 30 21:50:06.301437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.301401 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb" event={"ID":"e20632c7-f976-4027-b9ba-3478a2ca6b3e","Type":"ContainerDied","Data":"b1cd12fba6a321f45b88e768a1e2de629ece8c553160d4936f82408c6cc1f54b"} Oct 30 21:50:06.309521 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.309498 5036 scope.go:122] "RemoveContainer" containerID="869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893" Oct 30 21:50:06.309811 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:06.309788 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893\": container with ID starting with 869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893 not found: ID does not exist" containerID="869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893" Oct 30 21:50:06.309896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.309823 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893"} err="failed to get container status \"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893\": rpc error: code = NotFound desc = could not find container \"869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893\": container with ID starting with 869363a0daffcc9c5d631c9f9fbb9f24248582429c1afbeb9616b706f8e3a893 not found: ID does not exist" Oct 30 21:50:06.309896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.309846 5036 scope.go:122] "RemoveContainer" containerID="915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af" Oct 30 21:50:06.320247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.320216 5036 scope.go:122] "RemoveContainer" containerID="915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af" Oct 30 21:50:06.320595 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:06.320576 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af\": container with ID starting with 915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af not found: ID does not exist" containerID="915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af" Oct 30 21:50:06.320682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.320603 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af"} err="failed to get container status \"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af\": rpc error: code = NotFound desc = could not find container \"915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af\": container with ID starting with 915535f6fee720ee7fd6318e7ee3e5e33a01cfcaf06f15154c69a43216bbe4af not found: ID does not exist" Oct 30 21:50:06.336732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.336700 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:50:06.345737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.345707 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9528ebfc-696474b88c-qc4gb"] Oct 30 21:50:06.358962 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.358934 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:50:06.363114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:06.363089 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9528ebfc-967d9496c-592vb"] Oct 30 21:50:07.177703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.177671 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178252 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7aba5615-53f7-4f45-9291-2b492f558a82" containerName="payload-pre-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178285 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aba5615-53f7-4f45-9291-2b492f558a82" containerName="payload-pre-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178306 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" containerName="payload-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178312 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" containerName="payload-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178318 5036 cpu_manager.go:414] "containerMap: removing container" podUID="09974e4c-dbf6-46c5-b079-e50706be3871" containerName="revoke-keys" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178323 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09974e4c-dbf6-46c5-b079-e50706be3871" containerName="revoke-keys" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178337 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f8a890b0-9161-42c3-aeec-27918c58adfc" containerName="revoke-keys" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178341 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a890b0-9161-42c3-aeec-27918c58adfc" containerName="revoke-keys" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178451 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aba5615-53f7-4f45-9291-2b492f558a82" containerName="payload-pre-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178462 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" containerName="payload-processing" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178468 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="09974e4c-dbf6-46c5-b079-e50706be3871" containerName="revoke-keys" Oct 30 21:50:07.180182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.178476 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a890b0-9161-42c3-aeec-27918c58adfc" containerName="revoke-keys" Oct 30 21:50:07.181902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.181877 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.189831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.189483 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b2e72-openshift-default-dockercfg-98gbf\"" Oct 30 21:50:07.198704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.198677 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 21:50:07.274500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.274471 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:50:07.318762 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.318727 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5052911-0326-45e8-b13a-6687d3048d88" containerID="409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67" exitCode=0 Oct 30 21:50:07.318919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.318762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerDied","Data":"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67"} Oct 30 21:50:07.319299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.319246 5036 scope.go:122] "RemoveContainer" containerID="409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67" Oct 30 21:50:07.320718 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.320689 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.320819 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.320755 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.320819 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.320788 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.360826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.360799 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.360976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.360846 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.360976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.360921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.360976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.360939 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.361148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.360994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.361148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.361011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9r87\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.361148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.361029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.361148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.361049 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.361148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.361078 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.461929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.461840 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.461929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.461882 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.461946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.461967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9r87\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.461991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462211 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462387 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462628 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.462830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.462788 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.465345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.465315 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.465593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.465570 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.474838 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.474813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9r87\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.476359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.476338 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.527582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.527531 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:07.580680 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.580606 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:50:07.581068 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.580941 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" containerID="cri-o://a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d" gracePeriod=30 Oct 30 21:50:07.852363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.852172 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" podUID="4de95922-eed9-452e-a6dd-f02364b88916" containerName="payload-processing" containerID="cri-o://ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552" gracePeriod=25 Oct 30 21:50:07.861339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.861232 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" podUID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" containerName="payload-pre-processing" containerID="cri-o://533817533fad7b3d4c649dbf9222644ca6d3b25421c3435c4cdd670741e5a9cc" gracePeriod=25 Oct 30 21:50:07.876903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.876878 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 21:50:07.964296 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:07.964248 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2faffd2_41a1_48b4_9967_421734cd1ee3.slice/crio-ea00a84b07959903ff7a15b09374542eccb419ad2bf80bdfd0738e14d6c35fc3 WatchSource:0}: Error finding container ea00a84b07959903ff7a15b09374542eccb419ad2bf80bdfd0738e14d6c35fc3: Status 404 returned error can't find the container with id ea00a84b07959903ff7a15b09374542eccb419ad2bf80bdfd0738e14d6c35fc3 Oct 30 21:50:07.967914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.967879 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.967989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.967962 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.968047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:07.967999 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:08.193745 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.193713 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aba5615-53f7-4f45-9291-2b492f558a82" path="/var/lib/kubelet/pods/7aba5615-53f7-4f45-9291-2b492f558a82/volumes" Oct 30 21:50:08.194779 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.194755 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20632c7-f976-4027-b9ba-3478a2ca6b3e" path="/var/lib/kubelet/pods/e20632c7-f976-4027-b9ba-3478a2ca6b3e/volumes" Oct 30 21:50:08.198589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.198566 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:50:08.217233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.208611 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" podUID="94bebf2d-37b2-4892-b5c6-c79605e672f6" containerName="payload-processing" containerID="cri-o://99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080" gracePeriod=25 Oct 30 21:50:08.239735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.239700 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" podUID="a12c6010-be0d-4f4a-a872-c09001c44544" containerName="payload-pre-processing" containerID="cri-o://b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7" gracePeriod=25 Oct 30 21:50:08.342014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.340541 5036 generic.go:356] "Generic (PLEG): container finished" podID="4de95922-eed9-452e-a6dd-f02364b88916" containerID="ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552" exitCode=0 Oct 30 21:50:08.342014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.340677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" event={"ID":"4de95922-eed9-452e-a6dd-f02364b88916","Type":"ContainerDied","Data":"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552"} Oct 30 21:50:08.342014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.340736 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" event={"ID":"4de95922-eed9-452e-a6dd-f02364b88916","Type":"ContainerDied","Data":"0b5dfe667a16c7f223d3c0450297c951c0155a9c19a94203e4e59865285a1d53"} Oct 30 21:50:08.342014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.340770 5036 scope.go:122] "RemoveContainer" containerID="ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552" Oct 30 21:50:08.342014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.340929 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr" Oct 30 21:50:08.347168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.347141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerStarted","Data":"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b"} Oct 30 21:50:08.349364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.349332 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" event={"ID":"f38fb764-292a-4ce3-b1bc-cb3ce2456498","Type":"ContainerDied","Data":"533817533fad7b3d4c649dbf9222644ca6d3b25421c3435c4cdd670741e5a9cc"} Oct 30 21:50:08.350411 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.349182 5036 generic.go:356] "Generic (PLEG): container finished" podID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" containerID="533817533fad7b3d4c649dbf9222644ca6d3b25421c3435c4cdd670741e5a9cc" exitCode=0 Oct 30 21:50:08.353079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.353058 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerStarted","Data":"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba"} Oct 30 21:50:08.353246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.353227 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerStarted","Data":"ea00a84b07959903ff7a15b09374542eccb419ad2bf80bdfd0738e14d6c35fc3"} Oct 30 21:50:08.397082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.397060 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\" (UniqueName: \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\") pod \"4de95922-eed9-452e-a6dd-f02364b88916\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " Oct 30 21:50:08.397326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.397307 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\") pod \"4de95922-eed9-452e-a6dd-f02364b88916\" (UID: \"4de95922-eed9-452e-a6dd-f02364b88916\") " Oct 30 21:50:08.397665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.397643 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume" pod "4de95922-eed9-452e-a6dd-f02364b88916" (UID: "4de95922-eed9-452e-a6dd-f02364b88916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:08.399371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.399341 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl" pod "4de95922-eed9-452e-a6dd-f02364b88916" (UID: "4de95922-eed9-452e-a6dd-f02364b88916"). InnerVolumeSpecName "kube-api-access-4jxwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:08.441883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.424727 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podStartSLOduration=1.424703548 podStartE2EDuration="1.424703548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:08.410045374 +0000 UTC m=+632.360000332" watchObservedRunningTime="2026-10-30 21:50:08.424703548 +0000 UTC m=+632.374658494" Oct 30 21:50:08.464332 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.464301 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:50:08.465186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.465161 5036 scope.go:122] "RemoveContainer" containerID="ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552" Oct 30 21:50:08.465596 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:08.465572 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552\": container with ID starting with ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552 not found: ID does not exist" containerID="ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552" Oct 30 21:50:08.465655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.465612 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552"} err="failed to get container status \"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552\": rpc error: code = NotFound desc = could not find container \"ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552\": container with ID starting with ef62007cf47d3380c6a197449800939bfa2fc0cd1dce548e9a38a69e11049552 not found: ID does not exist" Oct 30 21:50:08.492291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.485217 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:50:08.492291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.486026 5036 patch_prober.go:29] interesting pod/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:50:08.492291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.486096 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:50:08.499875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.499845 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de95922-eed9-452e-a6dd-f02364b88916-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.500074 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.500059 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxwl\" (UniqueName: \"kubernetes.io/projected/4de95922-eed9-452e-a6dd-f02364b88916-kube-api-access-4jxwl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.530427 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.530395 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:08.647604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.647568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\") pod \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " Oct 30 21:50:08.647936 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.647912 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\" (UniqueName: \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\") pod \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\" (UID: \"f38fb764-292a-4ce3-b1bc-cb3ce2456498\") " Oct 30 21:50:08.648432 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.647989 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume" pod "f38fb764-292a-4ce3-b1bc-cb3ce2456498" (UID: "f38fb764-292a-4ce3-b1bc-cb3ce2456498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:08.657169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.655309 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98" pod "f38fb764-292a-4ce3-b1bc-cb3ce2456498" (UID: "f38fb764-292a-4ce3-b1bc-cb3ce2456498"). InnerVolumeSpecName "kube-api-access-pls98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:08.694522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.694490 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:50:08.698748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.698724 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-31a18d9b-7b9d995f5f-gnvtr"] Oct 30 21:50:08.701614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.701595 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:50:08.748831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.748778 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\") pod \"94bebf2d-37b2-4892-b5c6-c79605e672f6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " Oct 30 21:50:08.748831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.748826 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\" (UniqueName: \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\") pod \"94bebf2d-37b2-4892-b5c6-c79605e672f6\" (UID: \"94bebf2d-37b2-4892-b5c6-c79605e672f6\") " Oct 30 21:50:08.749135 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.749124 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38fb764-292a-4ce3-b1bc-cb3ce2456498-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.749135 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.749136 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pls98\" (UniqueName: \"kubernetes.io/projected/f38fb764-292a-4ce3-b1bc-cb3ce2456498-kube-api-access-pls98\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.749288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.749233 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume" pod "94bebf2d-37b2-4892-b5c6-c79605e672f6" (UID: "94bebf2d-37b2-4892-b5c6-c79605e672f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:08.752839 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.752792 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4" pod "94bebf2d-37b2-4892-b5c6-c79605e672f6" (UID: "94bebf2d-37b2-4892-b5c6-c79605e672f6"). InnerVolumeSpecName "kube-api-access-xkrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:08.807935 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.807887 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:50:08.850329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.849975 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\") pod \"a12c6010-be0d-4f4a-a872-c09001c44544\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " Oct 30 21:50:08.850502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.850428 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume" pod "a12c6010-be0d-4f4a-a872-c09001c44544" (UID: "a12c6010-be0d-4f4a-a872-c09001c44544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:08.850502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.850465 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bebf2d-37b2-4892-b5c6-c79605e672f6-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.850502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.850479 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrl4\" (UniqueName: \"kubernetes.io/projected/94bebf2d-37b2-4892-b5c6-c79605e672f6-kube-api-access-xkrl4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.956768 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.956662 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\" (UniqueName: \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\") pod \"a12c6010-be0d-4f4a-a872-c09001c44544\" (UID: \"a12c6010-be0d-4f4a-a872-c09001c44544\") " Oct 30 21:50:08.958377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.957535 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12c6010-be0d-4f4a-a872-c09001c44544-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:08.960543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.960508 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp" pod "a12c6010-be0d-4f4a-a872-c09001c44544" (UID: "a12c6010-be0d-4f4a-a872-c09001c44544"). InnerVolumeSpecName "kube-api-access-9rktp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:08.960887 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:08.960863 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:50:09.058289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.058153 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rktp\" (UniqueName: \"kubernetes.io/projected/a12c6010-be0d-4f4a-a872-c09001c44544-kube-api-access-9rktp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:09.361927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.361826 5036 generic.go:356] "Generic (PLEG): container finished" podID="94bebf2d-37b2-4892-b5c6-c79605e672f6" containerID="99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080" exitCode=0 Oct 30 21:50:09.361927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.361866 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" Oct 30 21:50:09.361927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.361900 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" event={"ID":"94bebf2d-37b2-4892-b5c6-c79605e672f6","Type":"ContainerDied","Data":"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080"} Oct 30 21:50:09.362502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.361961 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6" event={"ID":"94bebf2d-37b2-4892-b5c6-c79605e672f6","Type":"ContainerDied","Data":"14249677f88e93a5a3c4ef060676fca4abdffedf5ff7191fe251a25592388b1f"} Oct 30 21:50:09.362502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.361993 5036 scope.go:122] "RemoveContainer" containerID="99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080" Oct 30 21:50:09.363533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.363509 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" Oct 30 21:50:09.363644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.363574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb" event={"ID":"f38fb764-292a-4ce3-b1bc-cb3ce2456498","Type":"ContainerDied","Data":"caa48bb432205823c33772e8bfc68aea323760d9ae27f2ec9ff1febb13db1604"} Oct 30 21:50:09.366122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.366099 5036 generic.go:356] "Generic (PLEG): container finished" podID="a12c6010-be0d-4f4a-a872-c09001c44544" containerID="b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7" exitCode=0 Oct 30 21:50:09.366232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.366149 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" Oct 30 21:50:09.366232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.366187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" event={"ID":"a12c6010-be0d-4f4a-a872-c09001c44544","Type":"ContainerDied","Data":"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7"} Oct 30 21:50:09.366354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.366236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4" event={"ID":"a12c6010-be0d-4f4a-a872-c09001c44544","Type":"ContainerDied","Data":"6021908726c1b70214eafed098047bd72223984f4ad1f093145ed4ab7f20856c"} Oct 30 21:50:09.367500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.367471 5036 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" secret="" err="secret \"e2e-derive-4487d16a-openshift-default-dockercfg-gk4fx\" not found" Oct 30 21:50:09.381408 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.381363 5036 scope.go:122] "RemoveContainer" containerID="99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080" Oct 30 21:50:09.382067 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:09.382045 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080\": container with ID starting with 99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080 not found: ID does not exist" containerID="99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080" Oct 30 21:50:09.382168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.382082 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080"} err="failed to get container status \"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080\": rpc error: code = NotFound desc = could not find container \"99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080\": container with ID starting with 99cdb2903b4a1a4119013153aad0bad208fb8fc4febb400858e930f8dcbc3080 not found: ID does not exist" Oct 30 21:50:09.382168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.382098 5036 scope.go:122] "RemoveContainer" containerID="533817533fad7b3d4c649dbf9222644ca6d3b25421c3435c4cdd670741e5a9cc" Oct 30 21:50:09.391723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.391705 5036 scope.go:122] "RemoveContainer" containerID="b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7" Oct 30 21:50:09.400322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.400306 5036 scope.go:122] "RemoveContainer" containerID="b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7" Oct 30 21:50:09.400629 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:09.400608 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7\": container with ID starting with b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7 not found: ID does not exist" containerID="b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7" Oct 30 21:50:09.400712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.400635 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7"} err="failed to get container status \"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7\": rpc error: code = NotFound desc = could not find container \"b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7\": container with ID starting with b3b7c6a133deae0b85db85809137bd331f9fe739992ecf812a132858f8a391a7 not found: ID does not exist" Oct 30 21:50:09.401945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.401928 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:50:09.412446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.412406 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-367ad753-9566688cf-5gwj4"] Oct 30 21:50:09.448395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.448351 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:50:09.469146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.469123 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-31a18d9b-9dc6d948c-v4cjb"] Oct 30 21:50:09.484224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.484191 5036 patch_prober.go:29] interesting pod/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:50:09.484355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.484239 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:50:09.490814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.490790 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:50:09.496712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.496690 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-367ad753-b88bfb95f-2pvg6"] Oct 30 21:50:09.542043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:09.541671 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:10.187750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.187713 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de95922-eed9-452e-a6dd-f02364b88916" path="/var/lib/kubelet/pods/4de95922-eed9-452e-a6dd-f02364b88916/volumes" Oct 30 21:50:10.188123 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.188103 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bebf2d-37b2-4892-b5c6-c79605e672f6" path="/var/lib/kubelet/pods/94bebf2d-37b2-4892-b5c6-c79605e672f6/volumes" Oct 30 21:50:10.188488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.188472 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12c6010-be0d-4f4a-a872-c09001c44544" path="/var/lib/kubelet/pods/a12c6010-be0d-4f4a-a872-c09001c44544/volumes" Oct 30 21:50:10.188854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.188835 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" path="/var/lib/kubelet/pods/f38fb764-292a-4ce3-b1bc-cb3ce2456498/volumes" Oct 30 21:50:10.390743 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.390675 5036 generic.go:356] "Generic (PLEG): container finished" podID="621c551d-484c-46ca-bc34-d3d678a1538e" containerID="fdaa8ba7513c53d8e6ef447a8dc6e545dde9fb28cb839b8a25d6ee1bbf5307ec" exitCode=0 Oct 30 21:50:10.391586 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.391549 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" containerID="cri-o://04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b" gracePeriod=30 Oct 30 21:50:10.391876 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.391850 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerDied","Data":"fdaa8ba7513c53d8e6ef447a8dc6e545dde9fb28cb839b8a25d6ee1bbf5307ec"} Oct 30 21:50:10.392034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.392019 5036 scope.go:122] "RemoveContainer" containerID="5b1a5f46fb1c93ebdfd9dfa458245382abc6c3114d0dad166d48ec7d5a9143cd" Oct 30 21:50:10.395759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.395703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:10.400392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.400363 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:10.458885 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.458864 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:50:10.484087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484808 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484735 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484808 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484776 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.484913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.484882 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\") pod \"621c551d-484c-46ca-bc34-d3d678a1538e\" (UID: \"621c551d-484c-46ca-bc34-d3d678a1538e\") " Oct 30 21:50:10.485095 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485053 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.485239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485215 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.485311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:10.485369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485338 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.485491 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485469 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621c551d-484c-46ca-bc34-d3d678a1538e-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.485544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485500 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.485544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485514 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.485544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485539 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.485783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.485701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.487038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.487009 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "kube-api-access-62h28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:10.488018 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.487975 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:10.488337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.488305 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:10.489355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.489332 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy" pod "621c551d-484c-46ca-bc34-d3d678a1538e" (UID: "621c551d-484c-46ca-bc34-d3d678a1538e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.587110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.587078 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62h28\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-kube-api-access-62h28\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.587110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.587107 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621c551d-484c-46ca-bc34-d3d678a1538e-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.587110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.587116 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.587366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.587124 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621c551d-484c-46ca-bc34-d3d678a1538e-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.587366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.587133 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621c551d-484c-46ca-bc34-d3d678a1538e-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.980821 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.980796 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:50:10.990095 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990072 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990156 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990279 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990226 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990322 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990325 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.990495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990361 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990457 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990493 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.990600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990530 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\") pod \"afba3bb9-1211-4f56-890b-d82a6fc42d43\" (UID: \"afba3bb9-1211-4f56-890b-d82a6fc42d43\") " Oct 30 21:50:10.990753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:10.990892 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.990855 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.991433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.991406 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.991520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.991478 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.991520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.991500 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.991520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.991516 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afba3bb9-1211-4f56-890b-d82a6fc42d43-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.991682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.991415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.992721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.992677 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:10.992721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.992682 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:10.992837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.992735 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:10.992837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:10.992761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl" pod "afba3bb9-1211-4f56-890b-d82a6fc42d43" (UID: "afba3bb9-1211-4f56-890b-d82a6fc42d43"). InnerVolumeSpecName "kube-api-access-5cmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:11.095492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.092992 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.095492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.093031 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cmxl\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-kube-api-access-5cmxl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.095492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.093046 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.095492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.093061 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afba3bb9-1211-4f56-890b-d82a6fc42d43-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.095492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.093074 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afba3bb9-1211-4f56-890b-d82a6fc42d43-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.399340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.399306 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" Oct 30 21:50:11.399340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.399317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf" event={"ID":"621c551d-484c-46ca-bc34-d3d678a1538e","Type":"ContainerDied","Data":"8d14efd6866943fa416fbdc05d84d137a27c12b2bd2dd20a94da34dcd2798460"} Oct 30 21:50:11.399846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.399424 5036 scope.go:122] "RemoveContainer" containerID="fdaa8ba7513c53d8e6ef447a8dc6e545dde9fb28cb839b8a25d6ee1bbf5307ec" Oct 30 21:50:11.401435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.401406 5036 generic.go:356] "Generic (PLEG): container finished" podID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerID="1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa" exitCode=0 Oct 30 21:50:11.401542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.401466 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" Oct 30 21:50:11.401615 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.401559 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerDied","Data":"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa"} Oct 30 21:50:11.401615 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.401601 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7" event={"ID":"afba3bb9-1211-4f56-890b-d82a6fc42d43","Type":"ContainerDied","Data":"94b702636a40259b55c7e2b56687290a58838864a99ab20bd3ff082c8464c54d"} Oct 30 21:50:11.443948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.443920 5036 scope.go:122] "RemoveContainer" containerID="1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa" Oct 30 21:50:11.456577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.456556 5036 scope.go:122] "RemoveContainer" containerID="ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9" Oct 30 21:50:11.470030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.470005 5036 scope.go:122] "RemoveContainer" containerID="1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa" Oct 30 21:50:11.470639 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:11.470610 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa\": container with ID starting with 1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa not found: ID does not exist" containerID="1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa" Oct 30 21:50:11.470731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.470647 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa"} err="failed to get container status \"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa\": rpc error: code = NotFound desc = could not find container \"1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa\": container with ID starting with 1d75d979c85282af5e4a9238e9db9e8b38d601d3446eb78abd5838db05502cfa not found: ID does not exist" Oct 30 21:50:11.470731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.470668 5036 scope.go:122] "RemoveContainer" containerID="ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9" Oct 30 21:50:11.470975 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:11.470953 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9\": container with ID starting with ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9 not found: ID does not exist" containerID="ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9" Oct 30 21:50:11.471033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.470990 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9"} err="failed to get container status \"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9\": rpc error: code = NotFound desc = could not find container \"ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9\": container with ID starting with ceb5e51c6f724333dafbb0c9edf308d2b3b0bbd1f94480dd95ab24c664765dd9 not found: ID does not exist" Oct 30 21:50:11.485747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.485726 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:50:11.486502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.486470 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:50:11.503330 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.503310 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-367ad753-openshift-default-95cff747c-zqgx7"] Oct 30 21:50:11.564302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.564247 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:50:11.587458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.587423 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9528ebfc-openshift-default-787ff4cddf-d6rjf"] Oct 30 21:50:11.843228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843128 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 21:50:11.843908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843888 5036 cpu_manager.go:414] "containerMap: removing container" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.843967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843912 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.843967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843934 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" containerName="payload-pre-processing" Oct 30 21:50:11.843967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843943 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" containerName="payload-pre-processing" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843979 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4de95922-eed9-452e-a6dd-f02364b88916" containerName="payload-processing" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.843987 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de95922-eed9-452e-a6dd-f02364b88916" containerName="payload-processing" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844009 5036 cpu_manager.go:414] "containerMap: removing container" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844018 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844036 5036 cpu_manager.go:414] "containerMap: removing container" podUID="94bebf2d-37b2-4892-b5c6-c79605e672f6" containerName="payload-processing" Oct 30 21:50:11.844055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844043 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bebf2d-37b2-4892-b5c6-c79605e672f6" containerName="payload-processing" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844059 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a12c6010-be0d-4f4a-a872-c09001c44544" containerName="payload-pre-processing" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844066 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12c6010-be0d-4f4a-a872-c09001c44544" containerName="payload-pre-processing" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844078 5036 cpu_manager.go:414] "containerMap: removing container" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844085 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844193 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844207 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bebf2d-37b2-4892-b5c6-c79605e672f6" containerName="payload-processing" Oct 30 21:50:11.844223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844217 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12c6010-be0d-4f4a-a872-c09001c44544" containerName="payload-pre-processing" Oct 30 21:50:11.844454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844225 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" containerName="istio-proxy" Oct 30 21:50:11.844454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844237 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.844454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844247 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.844454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844257 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38fb764-292a-4ce3-b1bc-cb3ce2456498" containerName="payload-pre-processing" Oct 30 21:50:11.844454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.844292 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de95922-eed9-452e-a6dd-f02364b88916" containerName="payload-processing" Oct 30 21:50:11.849012 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.848986 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 21:50:11.849119 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.849106 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:11.849575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.849556 5036 cpu_manager.go:414] "containerMap: removing container" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.849657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.849578 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" containerName="istio-proxy" Oct 30 21:50:11.853194 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.853175 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:11.863888 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.863863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 21:50:11.864463 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.864432 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1529dc\"" Oct 30 21:50:11.867588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.867566 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1529dc-dockercfg-477rn\"" Oct 30 21:50:11.873923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.873897 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 21:50:11.900141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.900119 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:11.900247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.900197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:11.900325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.900305 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcxpr\" (UniqueName: \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:11.900360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:11.900343 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g562x\" (UniqueName: \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.001885 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.001849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.002072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.001936 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:12.002072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.001963 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcxpr\" (UniqueName: \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:12.002072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.001987 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g562x\" (UniqueName: \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.002676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.002652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:12.002765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.002652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.029282 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.029223 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcxpr\" (UniqueName: \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\") pod \"payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:12.030297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.030255 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g562x\" (UniqueName: \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\") pod \"payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.168618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.168580 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:12.179797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.179778 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:12.188624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.188595 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621c551d-484c-46ca-bc34-d3d678a1538e" path="/var/lib/kubelet/pods/621c551d-484c-46ca-bc34-d3d678a1538e/volumes" Oct 30 21:50:12.189132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.189112 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afba3bb9-1211-4f56-890b-d82a6fc42d43" path="/var/lib/kubelet/pods/afba3bb9-1211-4f56-890b-d82a6fc42d43/volumes" Oct 30 21:50:12.277293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.275356 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" podUID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" containerName="payload-pre-processing" containerID="cri-o://41452124dc8032e91478847482d5a1f01c71cdff57d02c5c491817e56773691a" gracePeriod=25 Oct 30 21:50:12.451887 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.451794 5036 generic.go:356] "Generic (PLEG): container finished" podID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" containerID="41452124dc8032e91478847482d5a1f01c71cdff57d02c5c491817e56773691a" exitCode=0 Oct 30 21:50:12.452968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.452775 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" event={"ID":"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5","Type":"ContainerDied","Data":"41452124dc8032e91478847482d5a1f01c71cdff57d02c5c491817e56773691a"} Oct 30 21:50:12.557043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.557006 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 21:50:12.557303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.557287 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 21:50:12.762005 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.761962 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:50:12.778717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.778689 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 21:50:12.779278 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.779245 5036 cpu_manager.go:414] "containerMap: removing container" podUID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" containerName="payload-pre-processing" Oct 30 21:50:12.779278 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.779278 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" containerName="payload-pre-processing" Oct 30 21:50:12.779421 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.779371 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" containerName="payload-pre-processing" Oct 30 21:50:12.783227 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.783208 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.808890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.808852 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ceb36a\"" Oct 30 21:50:12.860942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.859642 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 21:50:12.860942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.860096 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\" (UniqueName: \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\") pod \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " Oct 30 21:50:12.860942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.860190 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\") pod \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\" (UID: \"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5\") " Oct 30 21:50:12.860942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.860429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.860942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.860493 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8rh\" (UniqueName: \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.861746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.861677 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume" pod "555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" (UID: "555e9fdf-aa03-4fa3-8137-2b9e3b241ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:12.864940 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.864907 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 21:50:12.886388 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.886349 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl" pod "555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" (UID: "555e9fdf-aa03-4fa3-8137-2b9e3b241ad5"). InnerVolumeSpecName "kube-api-access-6sdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:12.887418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.887391 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:12.889108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.889082 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 21:50:12.898853 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.897594 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ceb36a-dockercfg-5dmt5\"" Oct 30 21:50:12.961379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961319 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.961493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961407 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8rh\" (UniqueName: \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.961553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961521 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:12.961600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961559 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s45p\" (UniqueName: \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:12.961705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961687 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdcl\" (UniqueName: \"kubernetes.io/projected/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-kube-api-access-6sdcl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:12.961831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.961819 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:12.962021 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.962003 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:12.966401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.966382 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:12.986974 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:12.986953 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8rh\" (UniqueName: \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\") pod \"payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071689 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071749 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071790 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071870 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071935 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.071969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.072014 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.072047 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.072078 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\") pod \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\" (UID: \"d549f0a4-3a6d-4e52-9037-0b12bac14b30\") " Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.072593 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:13.073473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.072644 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s45p\" (UniqueName: \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.079409 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.080228 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.080778 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "kube-api-access-zkl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.080935 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.081358 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.082288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.081541 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.082733 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.082693 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:13.083378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.083351 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.084900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.084865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.086666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.086645 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo" pod "d549f0a4-3a6d-4e52-9037-0b12bac14b30" (UID: "d549f0a4-3a6d-4e52-9037-0b12bac14b30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:13.099589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.099554 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s45p\" (UniqueName: \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\") pod \"payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:13.104635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.104607 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:13.174317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174252 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkl4t\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-kube-api-access-zkl4t\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174320 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174332 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174341 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174350 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174373 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174384 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174392 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.174488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.174401 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d549f0a4-3a6d-4e52-9037-0b12bac14b30-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.208361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.204237 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:13.463881 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:13.463851 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3930c8_8d17_4300_bef9_5cf75dc767e6.slice/crio-e87e63b63c15b04f9f4c5e2853a0466c76e68b9867acf6e43596411b2a9cd46b WatchSource:0}: Error finding container e87e63b63c15b04f9f4c5e2853a0466c76e68b9867acf6e43596411b2a9cd46b: Status 404 returned error can't find the container with id e87e63b63c15b04f9f4c5e2853a0466c76e68b9867acf6e43596411b2a9cd46b Oct 30 21:50:13.464941 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.464914 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 21:50:13.471730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.471140 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" event={"ID":"e8626147-8e13-47e6-81fa-29c398a5c843","Type":"ContainerStarted","Data":"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9"} Oct 30 21:50:13.471730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.471210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" event={"ID":"e8626147-8e13-47e6-81fa-29c398a5c843","Type":"ContainerStarted","Data":"8bb7b4d586bcfa81ee1f277168b0c667e4bc159aaef310771d0853c237c24a19"} Oct 30 21:50:13.471730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.471255 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:13.474312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.473426 5036 generic.go:356] "Generic (PLEG): container finished" podID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerID="a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d" exitCode=0 Oct 30 21:50:13.474312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.473516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerDied","Data":"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d"} Oct 30 21:50:13.474312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.473555 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" event={"ID":"d549f0a4-3a6d-4e52-9037-0b12bac14b30","Type":"ContainerDied","Data":"141bbb0a6271ae7ed3d6e42a33fb6ebf5f5a75fb4252942ed777bcdfcaff74f0"} Oct 30 21:50:13.474312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.473584 5036 scope.go:122] "RemoveContainer" containerID="a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d" Oct 30 21:50:13.474312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.473718 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s" Oct 30 21:50:13.478898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.478836 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" event={"ID":"1c29cd98-84db-4578-858e-e89b3de85bcd","Type":"ContainerStarted","Data":"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02"} Oct 30 21:50:13.478898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.478895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" event={"ID":"1c29cd98-84db-4578-858e-e89b3de85bcd","Type":"ContainerStarted","Data":"4912ed9be39293fc7c91b34bd8992a333984ec41a24693f7bafc8d24b6936f62"} Oct 30 21:50:13.479067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.478927 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:13.480735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.480716 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" Oct 30 21:50:13.480822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.480718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496" event={"ID":"555e9fdf-aa03-4fa3-8137-2b9e3b241ad5","Type":"ContainerDied","Data":"5d486547ed2b49ee6735e96d5cbe602c56a9e75a18264e8938830b17b940c552"} Oct 30 21:50:13.490486 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.490464 5036 scope.go:122] "RemoveContainer" containerID="843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98" Oct 30 21:50:13.498005 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.497967 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" podStartSLOduration=2.497954433 podStartE2EDuration="2.497954433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:13.497694851 +0000 UTC m=+637.447649796" watchObservedRunningTime="2026-10-30 21:50:13.497954433 +0000 UTC m=+637.447909378" Oct 30 21:50:13.514110 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.514088 5036 scope.go:122] "RemoveContainer" containerID="a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d" Oct 30 21:50:13.514647 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:13.514624 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d\": container with ID starting with a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d not found: ID does not exist" containerID="a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d" Oct 30 21:50:13.514814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.514782 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d"} err="failed to get container status \"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d\": rpc error: code = NotFound desc = could not find container \"a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d\": container with ID starting with a1884532bfb81e8a0bb05f6443db8666ecd41db90e1995fe03b3f22e4d22ce7d not found: ID does not exist" Oct 30 21:50:13.514911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.514817 5036 scope.go:122] "RemoveContainer" containerID="843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98" Oct 30 21:50:13.515159 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:13.515136 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98\": container with ID starting with 843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98 not found: ID does not exist" containerID="843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98" Oct 30 21:50:13.515250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.515167 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98"} err="failed to get container status \"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98\": rpc error: code = NotFound desc = could not find container \"843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98\": container with ID starting with 843be42aafdc1c385af3e55aa63731aa873b0e0fb02de0d475a2a31b1cd08d98 not found: ID does not exist" Oct 30 21:50:13.515250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.515185 5036 scope.go:122] "RemoveContainer" containerID="41452124dc8032e91478847482d5a1f01c71cdff57d02c5c491817e56773691a" Oct 30 21:50:13.541789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.541730 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" podStartSLOduration=2.541709435 podStartE2EDuration="2.541709435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:13.52347408 +0000 UTC m=+637.473429027" watchObservedRunningTime="2026-10-30 21:50:13.541709435 +0000 UTC m=+637.491664376" Oct 30 21:50:13.549906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.549880 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 21:50:13.550685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550621 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.550685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550644 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.550685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550661 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.550685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550668 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.550952 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550838 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.550952 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.550852 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" containerName="istio-proxy" Oct 30 21:50:13.575340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.569977 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 21:50:13.575340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.570942 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.575340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.574902 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 21:50:13.575340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.574953 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 21:50:13.575340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.575068 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.580379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.580346 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2b2e72\"" Oct 30 21:50:13.580534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.580479 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b2e72-dockercfg-ksmv8\"" Oct 30 21:50:13.587371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.587312 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:50:13.591157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.590854 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-681bcb24-openshift-default-f4c68f8f7-g6r8s"] Oct 30 21:50:13.614052 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.613009 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 21:50:13.625363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.625324 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:50:13.628769 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.628739 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4487d16a-6f4797dd95-jk496"] Oct 30 21:50:13.684168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.684132 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.684361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.684224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52srk\" (UniqueName: \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.684410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.684378 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rkm\" (UniqueName: \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.684487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.684471 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.785681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.785592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rkm\" (UniqueName: \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.785681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.785668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.785882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.785695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.785882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.785733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52srk\" (UniqueName: \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.786504 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.786483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.786599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.786579 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.804553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.804510 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rkm\" (UniqueName: \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\") pod \"payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:13.805343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.805320 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52srk\" (UniqueName: \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\") pod \"payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.904752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.904709 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:13.922666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:13.922630 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:14.188290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.188244 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555e9fdf-aa03-4fa3-8137-2b9e3b241ad5" path="/var/lib/kubelet/pods/555e9fdf-aa03-4fa3-8137-2b9e3b241ad5/volumes" Oct 30 21:50:14.188736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.188721 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d549f0a4-3a6d-4e52-9037-0b12bac14b30" path="/var/lib/kubelet/pods/d549f0a4-3a6d-4e52-9037-0b12bac14b30/volumes" Oct 30 21:50:14.200886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.200866 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 21:50:14.202973 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:14.202946 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9dd224_b3e8_49bd_af8c_191e1f9d5b6c.slice/crio-d0db9df36a8cb836f012b0ad84aa2e27d125781f3f3efe0cb588b79ebbfa9f03 WatchSource:0}: Error finding container d0db9df36a8cb836f012b0ad84aa2e27d125781f3f3efe0cb588b79ebbfa9f03: Status 404 returned error can't find the container with id d0db9df36a8cb836f012b0ad84aa2e27d125781f3f3efe0cb588b79ebbfa9f03 Oct 30 21:50:14.245019 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.244995 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 21:50:14.248633 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:14.248604 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c0f2c3_c3b2_47b1_97c7_ef2762218bf2.slice/crio-ff35d559d755bb18fd508a96e8e2010bb343ddac67cf57025c642d8234b31f6d WatchSource:0}: Error finding container ff35d559d755bb18fd508a96e8e2010bb343ddac67cf57025c642d8234b31f6d: Status 404 returned error can't find the container with id ff35d559d755bb18fd508a96e8e2010bb343ddac67cf57025c642d8234b31f6d Oct 30 21:50:14.490608 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.490517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" event={"ID":"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c","Type":"ContainerStarted","Data":"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545"} Oct 30 21:50:14.490608 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.490577 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" event={"ID":"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c","Type":"ContainerStarted","Data":"d0db9df36a8cb836f012b0ad84aa2e27d125781f3f3efe0cb588b79ebbfa9f03"} Oct 30 21:50:14.490608 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.490601 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:14.492259 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.492235 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" event={"ID":"1c3930c8-8d17-4300-bef9-5cf75dc767e6","Type":"ContainerStarted","Data":"2126f21b0eb19c25a332a0a68bbf511f0380ab1bc035d30d24d543d11b0477cb"} Oct 30 21:50:14.492398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.492380 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" event={"ID":"1c3930c8-8d17-4300-bef9-5cf75dc767e6","Type":"ContainerStarted","Data":"e87e63b63c15b04f9f4c5e2853a0466c76e68b9867acf6e43596411b2a9cd46b"} Oct 30 21:50:14.492453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.492415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:14.495870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.495839 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" event={"ID":"335e965e-3a7b-4dbc-9779-24f34148969a","Type":"ContainerStarted","Data":"825cbff6aa941e1d85716bab483284789a249718dcf69c964009e325b5f96d5d"} Oct 30 21:50:14.495990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.495880 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:14.495990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.495903 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" event={"ID":"335e965e-3a7b-4dbc-9779-24f34148969a","Type":"ContainerStarted","Data":"033d08e8857a07906679fc5568b6b1d1e8191f9a7b7fe35cdf3e3c92ce460ec5"} Oct 30 21:50:14.497472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.497449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" event={"ID":"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2","Type":"ContainerStarted","Data":"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa"} Oct 30 21:50:14.497567 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.497490 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" event={"ID":"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2","Type":"ContainerStarted","Data":"ff35d559d755bb18fd508a96e8e2010bb343ddac67cf57025c642d8234b31f6d"} Oct 30 21:50:14.497770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.497750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:14.535368 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.535318 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" podStartSLOduration=1.5353011460000001 podStartE2EDuration="1.535301146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.534067778 +0000 UTC m=+638.484022724" watchObservedRunningTime="2026-10-30 21:50:14.535301146 +0000 UTC m=+638.485256099" Oct 30 21:50:14.569423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.569370 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" podStartSLOduration=2.569352796 podStartE2EDuration="2.569352796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.567755605 +0000 UTC m=+638.517710551" watchObservedRunningTime="2026-10-30 21:50:14.569352796 +0000 UTC m=+638.519307744" Oct 30 21:50:14.597816 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.597795 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 21:50:14.601850 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.601831 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.603585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.603544 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" podStartSLOduration=2.6035303389999997 podStartE2EDuration="2.603530339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.600210129 +0000 UTC m=+638.550165075" watchObservedRunningTime="2026-10-30 21:50:14.603530339 +0000 UTC m=+638.553485284" Oct 30 21:50:14.609910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.609876 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 21:50:14.613708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.613688 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.616084 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.616058 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 21:50:14.618591 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.618561 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-939a16-dockercfg-rtm7j\"" Oct 30 21:50:14.636489 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.636458 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 21:50:14.637670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.637613 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" podStartSLOduration=1.637597065 podStartE2EDuration="1.637597065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.632536425 +0000 UTC m=+638.582491376" watchObservedRunningTime="2026-10-30 21:50:14.637597065 +0000 UTC m=+638.587552012" Oct 30 21:50:14.685527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.685497 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-939a16\"" Oct 30 21:50:14.694986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.694952 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.695090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.695021 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm89p\" (UniqueName: \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.695206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.695184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtc2\" (UniqueName: \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.695239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.695228 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.801094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.800990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtc2\" (UniqueName: \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.801094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.801050 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.801343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.801117 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.801343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.801166 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm89p\" (UniqueName: \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.802436 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.802385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.803205 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.803180 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.820847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.820813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm89p\" (UniqueName: \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\") pod \"payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.822588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.822550 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtc2\" (UniqueName: \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\") pod \"payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:14.918343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.918306 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:14.931185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:14.931151 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:15.245678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.245642 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 21:50:15.250170 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.250149 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 21:50:15.252030 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:15.252007 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a1e6f_330f_48ed_bac0_3265fe3e72c3.slice/crio-35a39c82fce0b1515029d8e976fb9dbfd6389107493279bbb16299f395e7b97a WatchSource:0}: Error finding container 35a39c82fce0b1515029d8e976fb9dbfd6389107493279bbb16299f395e7b97a: Status 404 returned error can't find the container with id 35a39c82fce0b1515029d8e976fb9dbfd6389107493279bbb16299f395e7b97a Oct 30 21:50:15.253354 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:15.253293 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12167c7e_c89c_412b_ac41_509c30db9551.slice/crio-00851ebfebfbc76cce64233106a819b803a65eca73a99a3c0b2ff6992f61a2ae WatchSource:0}: Error finding container 00851ebfebfbc76cce64233106a819b803a65eca73a99a3c0b2ff6992f61a2ae: Status 404 returned error can't find the container with id 00851ebfebfbc76cce64233106a819b803a65eca73a99a3c0b2ff6992f61a2ae Oct 30 21:50:15.505796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.505711 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" event={"ID":"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3","Type":"ContainerStarted","Data":"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548"} Oct 30 21:50:15.505796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.505777 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" event={"ID":"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3","Type":"ContainerStarted","Data":"35a39c82fce0b1515029d8e976fb9dbfd6389107493279bbb16299f395e7b97a"} Oct 30 21:50:15.506281 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.505951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:15.507487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.507459 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" event={"ID":"12167c7e-c89c-412b-ac41-509c30db9551","Type":"ContainerStarted","Data":"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894"} Oct 30 21:50:15.507603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.507509 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" event={"ID":"12167c7e-c89c-412b-ac41-509c30db9551","Type":"ContainerStarted","Data":"00851ebfebfbc76cce64233106a819b803a65eca73a99a3c0b2ff6992f61a2ae"} Oct 30 21:50:15.535641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.535593 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" podStartSLOduration=1.53557835 podStartE2EDuration="1.53557835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.53375815 +0000 UTC m=+639.483713095" watchObservedRunningTime="2026-10-30 21:50:15.53557835 +0000 UTC m=+639.485533296" Oct 30 21:50:15.562518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.562467 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" podStartSLOduration=1.562452771 podStartE2EDuration="1.562452771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.560366646 +0000 UTC m=+639.510321594" watchObservedRunningTime="2026-10-30 21:50:15.562452771 +0000 UTC m=+639.512407717" Oct 30 21:50:15.796351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.796241 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 21:50:15.800568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.800546 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 21:50:15.800684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.800658 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:15.805239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.805218 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:15.811677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.811655 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-0f26c5\"" Oct 30 21:50:15.819791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.819771 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 21:50:15.822766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.822745 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-0f26c5-dockercfg-l7cfj\"" Oct 30 21:50:15.823646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.823624 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 21:50:15.918367 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.918324 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4vs\" (UniqueName: \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:15.918543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.918455 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnkj\" (UniqueName: \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:15.918543 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.918507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:15.918798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:15.918584 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.036046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.036000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4vs\" (UniqueName: \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:16.036711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.036683 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnkj\" (UniqueName: \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.036931 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.036912 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:16.037067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.037049 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.056143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.037885 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:16.056542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.056507 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.066918 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.066891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4vs\" (UniqueName: \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\") pod \"payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:16.072872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.072839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnkj\" (UniqueName: \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\") pod \"payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.119254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.119228 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:16.131475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.131449 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:16.427008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.426963 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 21:50:16.431510 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:16.431482 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb600fc49_9c31_48b8_bace_345729c4897e.slice/crio-48bf3dfff662659cb87166bea4b6af8ad1f3dd905d6a45904e7c81cfa58c1543 WatchSource:0}: Error finding container 48bf3dfff662659cb87166bea4b6af8ad1f3dd905d6a45904e7c81cfa58c1543: Status 404 returned error can't find the container with id 48bf3dfff662659cb87166bea4b6af8ad1f3dd905d6a45904e7c81cfa58c1543 Oct 30 21:50:16.452716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.452687 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 21:50:16.518058 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.517531 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" event={"ID":"15f5b8cf-7ce4-44d8-ba21-244a250c6740","Type":"ContainerStarted","Data":"2c3b0eccfd54440a475ce218ae4f8b621f1177864086756827e8d428400aa8f0"} Oct 30 21:50:16.519081 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.519055 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" event={"ID":"b600fc49-9c31-48b8-bace-345729c4897e","Type":"ContainerStarted","Data":"48bf3dfff662659cb87166bea4b6af8ad1f3dd905d6a45904e7c81cfa58c1543"} Oct 30 21:50:16.520617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:16.520602 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:17.527766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.527727 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" event={"ID":"15f5b8cf-7ce4-44d8-ba21-244a250c6740","Type":"ContainerStarted","Data":"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534"} Oct 30 21:50:17.528290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.528137 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:17.529990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.529962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" event={"ID":"b600fc49-9c31-48b8-bace-345729c4897e","Type":"ContainerStarted","Data":"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052"} Oct 30 21:50:17.530099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.530020 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:17.551526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.551481 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" podStartSLOduration=2.551470196 podStartE2EDuration="2.551470196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.549988277 +0000 UTC m=+641.499943223" watchObservedRunningTime="2026-10-30 21:50:17.551470196 +0000 UTC m=+641.501425157" Oct 30 21:50:17.570081 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:17.570038 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" podStartSLOduration=2.570022923 podStartE2EDuration="2.570022923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.569472114 +0000 UTC m=+641.519427078" watchObservedRunningTime="2026-10-30 21:50:17.570022923 +0000 UTC m=+641.519977869" Oct 30 21:50:18.100256 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:18.100205 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:50:19.388526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.388491 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:50:19.474102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474075 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474260 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474127 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474260 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474158 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474260 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474198 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474311 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474327 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:19.474440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474371 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474430 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\") pod \"c5052911-0326-45e8-b13a-6687d3048d88\" (UID: \"c5052911-0326-45e8-b13a-6687d3048d88\") " Oct 30 21:50:19.474645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:19.474850 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:19.475036 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474976 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.475036 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474999 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5052911-0326-45e8-b13a-6687d3048d88-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.475178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.474972 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:19.475178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.475015 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.475178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.475029 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:19.476671 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.476649 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.476905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.476883 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "kube-api-access-sklwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.477022 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.477004 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:19.477074 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.477025 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo" pod "c5052911-0326-45e8-b13a-6687d3048d88" (UID: "c5052911-0326-45e8-b13a-6687d3048d88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:19.548923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.548888 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5052911-0326-45e8-b13a-6687d3048d88" containerID="04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b" exitCode=0 Oct 30 21:50:19.549075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.548951 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" Oct 30 21:50:19.549075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.548966 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerDied","Data":"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b"} Oct 30 21:50:19.549075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.549023 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z" event={"ID":"c5052911-0326-45e8-b13a-6687d3048d88","Type":"ContainerDied","Data":"03047db81c196909a1b562d0ab85ff68a342f456a0a38aa939a23c239c8f3197"} Oct 30 21:50:19.549075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.549045 5036 scope.go:122] "RemoveContainer" containerID="04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b" Oct 30 21:50:19.559678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.559661 5036 scope.go:122] "RemoveContainer" containerID="409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67" Oct 30 21:50:19.568939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.568916 5036 scope.go:122] "RemoveContainer" containerID="04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b" Oct 30 21:50:19.569184 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:19.569167 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b\": container with ID starting with 04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b not found: ID does not exist" containerID="04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b" Oct 30 21:50:19.569235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.569193 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b"} err="failed to get container status \"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b\": rpc error: code = NotFound desc = could not find container \"04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b\": container with ID starting with 04d221c00f9a45c50017e481edd21e6290f037e15bf80c413458982ec6ff3b5b not found: ID does not exist" Oct 30 21:50:19.569235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.569207 5036 scope.go:122] "RemoveContainer" containerID="409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67" Oct 30 21:50:19.569477 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:19.569462 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67\": container with ID starting with 409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67 not found: ID does not exist" containerID="409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67" Oct 30 21:50:19.569534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.569482 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67"} err="failed to get container status \"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67\": rpc error: code = NotFound desc = could not find container \"409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67\": container with ID starting with 409abc3e769c7e1df92ef392f1f09734c5d8715216dad9b402f0e7816f18ab67 not found: ID does not exist" Oct 30 21:50:19.576079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576058 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5052911-0326-45e8-b13a-6687d3048d88-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.576145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576081 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.576145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576091 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.576145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576100 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5052911-0326-45e8-b13a-6687d3048d88-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.576145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576110 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.576145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.576118 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklwn\" (UniqueName: \"kubernetes.io/projected/c5052911-0326-45e8-b13a-6687d3048d88-kube-api-access-sklwn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.616248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.616217 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:50:19.620010 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:19.619979 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4487d16a-openshift-default-c8d985d57-94r2z"] Oct 30 21:50:20.187606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:20.187574 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5052911-0326-45e8-b13a-6687d3048d88" path="/var/lib/kubelet/pods/c5052911-0326-45e8-b13a-6687d3048d88/volumes" Oct 30 21:50:24.501244 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:24.501214 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 21:50:24.501676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:24.501303 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 21:50:25.522725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:25.521677 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 21:50:25.532462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:25.532433 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 21:50:25.541292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:25.533418 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 21:50:25.545450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:25.545431 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 21:50:26.523381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:26.523349 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 21:50:27.534061 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.534031 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.866332 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.866936 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.866955 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.866999 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.867006 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.867128 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.868315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.867142 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5052911-0326-45e8-b13a-6687d3048d88" containerName="istio-proxy" Oct 30 21:50:27.871655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.871620 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.877997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.877802 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"default-dockercfg-86s4f\"" Oct 30 21:50:27.877997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.877930 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"simulator-w3-ceb36a-kserve-self-signed-certs\"" Oct 30 21:50:27.882354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.881827 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 21:50:27.898813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.898784 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.898929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.898829 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.898974 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.898957 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.899027 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.899010 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.899125 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.899099 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5qm\" (UniqueName: \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.899239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.899224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.950417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.950383 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"openshift-service-ca.crt\"" Oct 30 21:50:27.950417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.950393 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"kube-root-ca.crt\"" Oct 30 21:50:27.999852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.999794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:27.999852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.999833 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5qm\" (UniqueName: \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.000097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.999943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.000097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:27.999986 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.000097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.000013 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.000097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.000047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.000326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.000304 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.001090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.000973 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.001090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.001039 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.003027 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.003007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.003318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.003299 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.013192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.013164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5qm\" (UniqueName: \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\") pod \"simulator-w3-ceb36a-kserve-689b67bc4d-m2lws\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.186649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.186616 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:28.479062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.479029 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 21:50:28.479307 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:28.479034 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787ab643_d777_4294_8bf2_7b3087c6ff02.slice/crio-1f404080e266c55e30c4b335445d5bddec444c7a67038b553936a2bbe4143b91 WatchSource:0}: Error finding container 1f404080e266c55e30c4b335445d5bddec444c7a67038b553936a2bbe4143b91: Status 404 returned error can't find the container with id 1f404080e266c55e30c4b335445d5bddec444c7a67038b553936a2bbe4143b91 Oct 30 21:50:28.542758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.542735 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 21:50:28.543098 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.542801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 21:50:28.618576 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.618543 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" event={"ID":"787ab643-d777-4294-8bf2-7b3087c6ff02","Type":"ContainerStarted","Data":"6b78051d8be87faae2c07a5abbc4b1b95ee38cdef9f0bb2a022953123a9d5a4d"} Oct 30 21:50:28.618796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.618585 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" event={"ID":"787ab643-d777-4294-8bf2-7b3087c6ff02","Type":"ContainerStarted","Data":"1f404080e266c55e30c4b335445d5bddec444c7a67038b553936a2bbe4143b91"} Oct 30 21:50:28.640115 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:28.640066 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podStartSLOduration=1.640052525 podStartE2EDuration="1.640052525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:28.638116723 +0000 UTC m=+652.588071670" watchObservedRunningTime="2026-10-30 21:50:28.640052525 +0000 UTC m=+652.590007471" Oct 30 21:50:29.186934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.186900 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:29.408377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.408340 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 21:50:29.413127 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.413102 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.426126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.426096 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"kube-root-ca.crt\"" Oct 30 21:50:29.426380 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.426234 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"openshift-service-ca.crt\"" Oct 30 21:50:29.426857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.426817 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"default-dockercfg-brj97\"" Oct 30 21:50:29.426857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.426817 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"simulator-w2-2b2e72-kserve-self-signed-certs\"" Oct 30 21:50:29.432813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.432791 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 21:50:29.549071 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.548990 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.549071 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.549046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.549535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.549140 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.549535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.549246 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.549535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.549306 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.549535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.549342 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.651077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651046 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651112 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651307 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651361 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651598 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.651773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.652033 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.653908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.653882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.654241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.654219 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.664803 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.664778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\") pod \"simulator-w2-2b2e72-kserve-76ffd77749-wdgfj\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:29.735389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:29.735354 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:30.046141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.046116 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 21:50:30.047878 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:30.047854 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e9233a1_3653_4889_a319_1dda932f5c0a.slice/crio-21a1d10bcd1e390c15d27516120939902537f03328734e60587eb766cae43a11 WatchSource:0}: Error finding container 21a1d10bcd1e390c15d27516120939902537f03328734e60587eb766cae43a11: Status 404 returned error can't find the container with id 21a1d10bcd1e390c15d27516120939902537f03328734e60587eb766cae43a11 Oct 30 21:50:30.638418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.638382 5036 generic.go:356] "Generic (PLEG): container finished" podID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerID="362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba" exitCode=0 Oct 30 21:50:30.638866 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.638469 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerDied","Data":"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba"} Oct 30 21:50:30.638983 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.638966 5036 scope.go:122] "RemoveContainer" containerID="362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba" Oct 30 21:50:30.640456 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.640430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" event={"ID":"5e9233a1-3653-4889-a319-1dda932f5c0a","Type":"ContainerStarted","Data":"065b6d83a6fe6230f4a7ec1fab41c230cc60271ca5d90bbfa068ae7cc0f4ca58"} Oct 30 21:50:30.640573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.640467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" event={"ID":"5e9233a1-3653-4889-a319-1dda932f5c0a","Type":"ContainerStarted","Data":"21a1d10bcd1e390c15d27516120939902537f03328734e60587eb766cae43a11"} Oct 30 21:50:30.640573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.640534 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:30.640661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.640623 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:30.640702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.640684 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:30.737150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:30.737123 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:31.649323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:31.649258 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerStarted","Data":"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820"} Oct 30 21:50:31.739796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:31.739745 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podStartSLOduration=2.73973105 podStartE2EDuration="2.73973105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:30.702461687 +0000 UTC m=+654.652416648" watchObservedRunningTime="2026-10-30 21:50:31.73973105 +0000 UTC m=+655.689685996" Oct 30 21:50:32.534668 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:32.534634 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:32.535008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:32.534986 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:50:32.535171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:32.535146 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:50:33.531289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:33.530666 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:50:33.531289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:33.530730 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:50:34.533660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:34.533620 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:50:34.534326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:34.534256 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:50:35.532033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:35.531584 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:50:35.532033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:35.531657 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:50:36.531118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:36.531026 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:50:36.531118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:36.531089 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:50:37.539943 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:37.539913 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:37.540387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:37.540366 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:37.541250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:37.541230 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 21:50:38.191530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:38.191489 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:38.201712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:38.201690 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:38.724133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:38.724100 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:50:39.736724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:39.736690 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:39.749159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:39.749135 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:40.740300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:40.740251 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:50:48.099703 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:50:48.099617 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:50:48.799763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.799729 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 21:50:48.805669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.805646 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.814171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.814148 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"premium-w2-2b2e72-kserve-self-signed-certs\"" Oct 30 21:50:48.815306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.815248 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 21:50:48.895004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.894975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.895193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.895030 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.895193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.895186 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.895355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.895216 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l24w\" (UniqueName: \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.895355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.895317 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.895506 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.895489 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996548 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996595 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l24w\" (UniqueName: \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.996643 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.997173 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.997419 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:48.998285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:48.997463 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:49.000589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.000493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:49.002308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.002283 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:49.011296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.011256 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l24w\" (UniqueName: \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\") pod \"premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:49.120361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.120258 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:49.416604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.416576 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 21:50:49.419006 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:50:49.418970 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf80e61a3_945b_4f1f_a486_3c93cffb3a64.slice/crio-8170018d05757ed1515d1c90ad4a3b06dad4f2a1ca8c4d60ab1e11e6952c14b9 WatchSource:0}: Error finding container 8170018d05757ed1515d1c90ad4a3b06dad4f2a1ca8c4d60ab1e11e6952c14b9: Status 404 returned error can't find the container with id 8170018d05757ed1515d1c90ad4a3b06dad4f2a1ca8c4d60ab1e11e6952c14b9 Oct 30 21:50:49.793381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.793291 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" event={"ID":"f80e61a3-945b-4f1f-a486-3c93cffb3a64","Type":"ContainerStarted","Data":"eeb7338ecc893527a391a6ea831c335eee9a3007865f6f8b343f444024b843f5"} Oct 30 21:50:49.793381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.793337 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" event={"ID":"f80e61a3-945b-4f1f-a486-3c93cffb3a64","Type":"ContainerStarted","Data":"8170018d05757ed1515d1c90ad4a3b06dad4f2a1ca8c4d60ab1e11e6952c14b9"} Oct 30 21:50:49.818832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:49.818785 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podStartSLOduration=1.818767204 podStartE2EDuration="1.818767204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:49.81634751 +0000 UTC m=+673.766302457" watchObservedRunningTime="2026-10-30 21:50:49.818767204 +0000 UTC m=+673.768722178" Oct 30 21:50:50.121232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:50.121140 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:59.121355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:59.121314 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:59.139600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:59.139570 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:50:59.883215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:50:59.883185 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:51:04.982075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.982042 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 21:51:04.987432 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.987409 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:04.994363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.993911 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"premium-w4-939a16-kserve-self-signed-certs\"" Oct 30 21:51:04.995526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.994406 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"default-dockercfg-tsrf9\"" Oct 30 21:51:04.995526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.994483 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"openshift-service-ca.crt\"" Oct 30 21:51:04.995526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:04.994550 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"kube-root-ca.crt\"" Oct 30 21:51:05.000539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.000479 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 21:51:05.078002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.077967 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.078171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.078018 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.078171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.078136 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.078283 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.078183 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.078373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.078351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.078428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.078412 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7562\" (UniqueName: \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179452 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179556 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179602 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7562\" (UniqueName: \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179672 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179928 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.179992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.179972 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.180097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.180022 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.182570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.182541 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.182676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.182642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.192847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.192818 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7562\" (UniqueName: \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\") pod \"premium-w4-939a16-kserve-68655b5949-qch5s\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.306064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.305983 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:05.593679 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.593473 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 21:51:05.596708 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:05.596670 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08d8a0d7_4085_40f2_a750_d0c5d49b3b0a.slice/crio-1bf3347df3cb48f185d373b5f245315d439542104ac6883fbc1f933135ac7eec WatchSource:0}: Error finding container 1bf3347df3cb48f185d373b5f245315d439542104ac6883fbc1f933135ac7eec: Status 404 returned error can't find the container with id 1bf3347df3cb48f185d373b5f245315d439542104ac6883fbc1f933135ac7eec Oct 30 21:51:05.915718 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.915675 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" event={"ID":"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a","Type":"ContainerStarted","Data":"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25"} Oct 30 21:51:05.915900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.915729 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" event={"ID":"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a","Type":"ContainerStarted","Data":"1bf3347df3cb48f185d373b5f245315d439542104ac6883fbc1f933135ac7eec"} Oct 30 21:51:05.935725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:05.935678 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podStartSLOduration=1.935664793 podStartE2EDuration="1.935664793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:05.935065536 +0000 UTC m=+689.885020482" watchObservedRunningTime="2026-10-30 21:51:05.935664793 +0000 UTC m=+689.885619822" Oct 30 21:51:06.306705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:06.306625 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:08.427118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.427079 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 21:51:08.432165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.432141 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.442378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.442352 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"default-dockercfg-ck8pq\"" Oct 30 21:51:08.442507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.442404 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"kube-root-ca.crt\"" Oct 30 21:51:08.442507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.442448 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"premium-w1-0f26c5-kserve-self-signed-certs\"" Oct 30 21:51:08.442677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.442632 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"openshift-service-ca.crt\"" Oct 30 21:51:08.447323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.446758 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 21:51:08.522157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522106 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.522374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.522374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522357 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.522513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522399 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7jg\" (UniqueName: \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.522513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.522513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.522451 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623727 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7jg\" (UniqueName: \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623757 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623775 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623842 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.623884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.623885 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.624192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.624141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.624247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.624198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.624313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.624292 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.626532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.626502 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.626874 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.626854 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.636683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.636662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7jg\" (UniqueName: \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\") pod \"premium-w1-0f26c5-kserve-6c898c44b7-jzwvg\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:08.758502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:08.758424 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:09.043848 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:09.043814 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 21:51:09.046318 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:09.046289 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad4a8350_47f0_480a_9ac3_6b993bd07c1e.slice/crio-c2a8de16a9fa66adf23bd325d715e254e7d90c513a50babcf4b9a8188de19a56 WatchSource:0}: Error finding container c2a8de16a9fa66adf23bd325d715e254e7d90c513a50babcf4b9a8188de19a56: Status 404 returned error can't find the container with id c2a8de16a9fa66adf23bd325d715e254e7d90c513a50babcf4b9a8188de19a56 Oct 30 21:51:09.947038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:09.946993 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" event={"ID":"ad4a8350-47f0-480a-9ac3-6b993bd07c1e","Type":"ContainerStarted","Data":"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3"} Oct 30 21:51:09.947609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:09.947053 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" event={"ID":"ad4a8350-47f0-480a-9ac3-6b993bd07c1e","Type":"ContainerStarted","Data":"c2a8de16a9fa66adf23bd325d715e254e7d90c513a50babcf4b9a8188de19a56"} Oct 30 21:51:10.759552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:10.759517 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:15.307133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:15.307105 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:15.320527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:15.320493 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:15.367230 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:15.367180 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podStartSLOduration=7.367166956 podStartE2EDuration="7.367166956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:09.970008074 +0000 UTC m=+693.919963016" watchObservedRunningTime="2026-10-30 21:51:15.367166956 +0000 UTC m=+699.317121902" Oct 30 21:51:15.998934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:15.998905 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:51:18.098367 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:51:18.098257 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:51:18.758849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:18.758813 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:18.773302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:18.773257 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:19.018205 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:19.018120 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:51:26.793954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.793916 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf"] Oct 30 21:51:26.798595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.798576 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.803077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.803048 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:26.803189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.803080 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:26.803851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.803828 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:26.803955 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.803890 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:51:26.807523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.807502 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf"] Oct 30 21:51:26.822140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.822101 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45g4\" (UniqueName: \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.822294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.822155 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.822294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.822240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.923522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.923485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45g4\" (UniqueName: \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.923716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.923531 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.923716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.923575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.924322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.924294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.926399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.926378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:26.936142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:26.936115 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45g4\" (UniqueName: \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\") pod \"maas-api-revoke-sub-48902a9d-jl9cf\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:27.116423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:27.116322 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:27.417369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:27.417341 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf"] Oct 30 21:51:27.419133 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:27.419103 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c9727f_099a_46ac_9814_2e23e7e53bc2.slice/crio-b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96 WatchSource:0}: Error finding container b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96: Status 404 returned error can't find the container with id b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96 Oct 30 21:51:28.080582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:28.080490 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" event={"ID":"27c9727f-099a-46ac-9814-2e23e7e53bc2","Type":"ContainerStarted","Data":"fc4c40aff4a42754ed9ae7fa408eb9d794c8a9f0bd311a87ea938a107668ebf2"} Oct 30 21:51:28.080582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:28.080570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" event={"ID":"27c9727f-099a-46ac-9814-2e23e7e53bc2","Type":"ContainerStarted","Data":"b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96"} Oct 30 21:51:28.098583 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:28.098537 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" podStartSLOduration=2.09852489 podStartE2EDuration="2.09852489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:28.098135338 +0000 UTC m=+712.048090283" watchObservedRunningTime="2026-10-30 21:51:28.09852489 +0000 UTC m=+712.048479835" Oct 30 21:51:29.088923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:29.088877 5036 generic.go:356] "Generic (PLEG): container finished" podID="27c9727f-099a-46ac-9814-2e23e7e53bc2" containerID="fc4c40aff4a42754ed9ae7fa408eb9d794c8a9f0bd311a87ea938a107668ebf2" exitCode=0 Oct 30 21:51:29.088923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:29.088915 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" event={"ID":"27c9727f-099a-46ac-9814-2e23e7e53bc2","Type":"ContainerDied","Data":"fc4c40aff4a42754ed9ae7fa408eb9d794c8a9f0bd311a87ea938a107668ebf2"} Oct 30 21:51:30.286755 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.286731 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:30.363317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.363203 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\") pod \"27c9727f-099a-46ac-9814-2e23e7e53bc2\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " Oct 30 21:51:30.363534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.363399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\" (UniqueName: \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\") pod \"27c9727f-099a-46ac-9814-2e23e7e53bc2\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " Oct 30 21:51:30.363534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.363458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\") pod \"27c9727f-099a-46ac-9814-2e23e7e53bc2\" (UID: \"27c9727f-099a-46ac-9814-2e23e7e53bc2\") " Oct 30 21:51:30.363725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.363597 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca" pod "27c9727f-099a-46ac-9814-2e23e7e53bc2" (UID: "27c9727f-099a-46ac-9814-2e23e7e53bc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:30.363913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.363888 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:30.366335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.366306 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4" pod "27c9727f-099a-46ac-9814-2e23e7e53bc2" (UID: "27c9727f-099a-46ac-9814-2e23e7e53bc2"). InnerVolumeSpecName "kube-api-access-b45g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:30.367283 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.367239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca" pod "27c9727f-099a-46ac-9814-2e23e7e53bc2" (UID: "27c9727f-099a-46ac-9814-2e23e7e53bc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:30.464746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.464712 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45g4\" (UniqueName: \"kubernetes.io/projected/27c9727f-099a-46ac-9814-2e23e7e53bc2-kube-api-access-b45g4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:30.464746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:30.464744 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c9727f-099a-46ac-9814-2e23e7e53bc2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.106004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:31.105964 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" event={"ID":"27c9727f-099a-46ac-9814-2e23e7e53bc2","Type":"ContainerDied","Data":"b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96"} Oct 30 21:51:31.106004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:31.105986 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf" Oct 30 21:51:31.106219 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:31.106027 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b375bfdae44ff12a667d860f6778ea5068635f9acaae6574208936fb4720db96" Oct 30 21:51:36.625334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.625296 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 21:51:36.626127 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.626105 5036 cpu_manager.go:414] "containerMap: removing container" podUID="27c9727f-099a-46ac-9814-2e23e7e53bc2" containerName="revoke-keys" Oct 30 21:51:36.626224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.626129 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c9727f-099a-46ac-9814-2e23e7e53bc2" containerName="revoke-keys" Oct 30 21:51:36.626325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.626311 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c9727f-099a-46ac-9814-2e23e7e53bc2" containerName="revoke-keys" Oct 30 21:51:36.630173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.630150 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.640362 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.639960 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 21:51:36.642815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.642790 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1529dc-openshift-default-dockercfg-zmbwx\"" Oct 30 21:51:36.736710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.736673 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.736710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.736714 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.736968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.736734 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.736968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.736828 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.736968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.736872 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48w4\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.737133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.737027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.737133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.737075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.737236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.737135 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.737236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.737197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838024 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.837987 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838043 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838218 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838246 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838283 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48w4\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838429 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838659 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838470 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838659 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838541 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838659 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838865 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.838948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.838877 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.839028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.839010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.841180 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.841160 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.841336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.841308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.850901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.850876 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.851124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.851102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48w4\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:36.950475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:36.950436 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:37.245630 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:37.245598 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 21:51:37.247476 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:37.247450 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194dc5f4_757c_422b_a4ee_ac0482d8b0e4.slice/crio-4c53a6de80352e618da9bb281e3356b46f42905018f2abe9d8954cf7489348f1 WatchSource:0}: Error finding container 4c53a6de80352e618da9bb281e3356b46f42905018f2abe9d8954cf7489348f1: Status 404 returned error can't find the container with id 4c53a6de80352e618da9bb281e3356b46f42905018f2abe9d8954cf7489348f1 Oct 30 21:51:37.250459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:37.250419 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:37.250566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:37.250504 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:37.250566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:37.250549 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:38.160212 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.160171 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" event={"ID":"194dc5f4-757c-422b-a4ee-ac0482d8b0e4","Type":"ContainerStarted","Data":"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89"} Oct 30 21:51:38.160737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.160237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" event={"ID":"194dc5f4-757c-422b-a4ee-ac0482d8b0e4","Type":"ContainerStarted","Data":"4c53a6de80352e618da9bb281e3356b46f42905018f2abe9d8954cf7489348f1"} Oct 30 21:51:38.185316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.185245 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podStartSLOduration=2.185230513 podStartE2EDuration="2.185230513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:38.181816966 +0000 UTC m=+722.131771913" watchObservedRunningTime="2026-10-30 21:51:38.185230513 +0000 UTC m=+722.135185459" Oct 30 21:51:38.950918 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.950882 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:38.952918 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.952894 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:51:38.953040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:38.952940 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:51:39.951211 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:39.951175 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:51:39.951644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:39.951229 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:51:39.996424 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:39.996394 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx"] Oct 30 21:51:40.000549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.000521 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.004867 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.004845 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:40.007800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.007779 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:40.008436 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.008412 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:40.011164 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.010400 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:51:40.048311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.044356 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx"] Oct 30 21:51:40.075518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.075479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.075678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.075525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlnh\" (UniqueName: \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.075678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.075633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.176379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.176349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.176553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.176391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlnh\" (UniqueName: \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.176553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.176426 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.177052 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.177028 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.178926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.178897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.190198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.190174 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlnh\" (UniqueName: \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\") pod \"maas-api-revoke-sub-8b4b17fb-tvpsx\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.330339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.329475 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:40.644313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.642663 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx"] Oct 30 21:51:40.645693 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:40.645667 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08153fd3_a1bb_43d5_9643_ba875cf0d511.slice/crio-6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56 WatchSource:0}: Error finding container 6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56: Status 404 returned error can't find the container with id 6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56 Oct 30 21:51:40.951040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.951010 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:51:40.951179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:40.951069 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:51:41.183148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:41.183114 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" event={"ID":"08153fd3-a1bb-43d5-9643-ba875cf0d511","Type":"ContainerStarted","Data":"61e122adff745990685646b140db9075607c2a9e3371b09be8bcb2c4306d26f7"} Oct 30 21:51:41.183647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:41.183188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" event={"ID":"08153fd3-a1bb-43d5-9643-ba875cf0d511","Type":"ContainerStarted","Data":"6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56"} Oct 30 21:51:41.203548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:41.203459 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" podStartSLOduration=2.203446076 podStartE2EDuration="2.203446076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:41.200634161 +0000 UTC m=+725.150589122" watchObservedRunningTime="2026-10-30 21:51:41.203446076 +0000 UTC m=+725.153401379" Oct 30 21:51:41.951657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:41.951626 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:51:41.951810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:41.951688 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:51:42.191247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.191210 5036 generic.go:356] "Generic (PLEG): container finished" podID="08153fd3-a1bb-43d5-9643-ba875cf0d511" containerID="61e122adff745990685646b140db9075607c2a9e3371b09be8bcb2c4306d26f7" exitCode=0 Oct 30 21:51:42.191721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.191306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" event={"ID":"08153fd3-a1bb-43d5-9643-ba875cf0d511","Type":"ContainerDied","Data":"61e122adff745990685646b140db9075607c2a9e3371b09be8bcb2c4306d26f7"} Oct 30 21:51:42.683997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.683958 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2"] Oct 30 21:51:42.688017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.688000 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.698606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.698580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2"] Oct 30 21:51:42.849970 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.849927 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.850191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.850111 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.850316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.850213 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wz8\" (UniqueName: \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.950955 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.950862 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:51:42.951133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.950965 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:51:42.951133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.950981 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wz8\" (UniqueName: \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.951133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.951067 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.951293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.951159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.952060 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.952032 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.954227 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.954203 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:42.963607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:42.963586 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wz8\" (UniqueName: \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\") pod \"maas-api-revoke-sub-8a23a8ee-95qv2\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:43.003682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.003660 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:43.321474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.321350 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2"] Oct 30 21:51:43.383399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.383374 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:43.558671 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.558583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\") pod \"08153fd3-a1bb-43d5-9643-ba875cf0d511\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " Oct 30 21:51:43.558671 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.558626 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\") pod \"08153fd3-a1bb-43d5-9643-ba875cf0d511\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " Oct 30 21:51:43.558915 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.558693 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\" (UniqueName: \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\") pod \"08153fd3-a1bb-43d5-9643-ba875cf0d511\" (UID: \"08153fd3-a1bb-43d5-9643-ba875cf0d511\") " Oct 30 21:51:43.559185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.559148 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca" pod "08153fd3-a1bb-43d5-9643-ba875cf0d511" (UID: "08153fd3-a1bb-43d5-9643-ba875cf0d511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:43.561656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.561628 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca" pod "08153fd3-a1bb-43d5-9643-ba875cf0d511" (UID: "08153fd3-a1bb-43d5-9643-ba875cf0d511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:43.561749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.561659 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh" pod "08153fd3-a1bb-43d5-9643-ba875cf0d511" (UID: "08153fd3-a1bb-43d5-9643-ba875cf0d511"). InnerVolumeSpecName "kube-api-access-nqlnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:43.659878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.659846 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.659878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.659872 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08153fd3-a1bb-43d5-9643-ba875cf0d511-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.659878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:43.659883 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlnh\" (UniqueName: \"kubernetes.io/projected/08153fd3-a1bb-43d5-9643-ba875cf0d511-kube-api-access-nqlnh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.042941 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.042909 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:44.043914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.043889 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:44.046364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.046328 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 21:51:44.207305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.207228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" event={"ID":"61634f8d-5849-4ef1-a362-f1d093ba1154","Type":"ContainerStarted","Data":"b9a30e457912ce8b7c2909e32784c7116304b8b55a754d7e5877620a89da53d5"} Oct 30 21:51:44.207460 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.207318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" event={"ID":"61634f8d-5849-4ef1-a362-f1d093ba1154","Type":"ContainerStarted","Data":"2ce32827735727596ce52a96dcec88571392a40a2fc0d623c99595d7ba76f10a"} Oct 30 21:51:44.208913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.208889 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" event={"ID":"08153fd3-a1bb-43d5-9643-ba875cf0d511","Type":"ContainerDied","Data":"6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56"} Oct 30 21:51:44.209059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.208918 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx" Oct 30 21:51:44.209059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.208927 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6815405d6360b2b8bb5f66787271d1c33bd3ef4b01403fbdc91cdfff6d96cd56" Oct 30 21:51:44.233058 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:44.233002 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" podStartSLOduration=2.232982483 podStartE2EDuration="2.232982483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:44.230399375 +0000 UTC m=+728.180354321" watchObservedRunningTime="2026-10-30 21:51:44.232982483 +0000 UTC m=+728.182937430" Oct 30 21:51:45.217642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:45.217610 5036 generic.go:356] "Generic (PLEG): container finished" podID="61634f8d-5849-4ef1-a362-f1d093ba1154" containerID="b9a30e457912ce8b7c2909e32784c7116304b8b55a754d7e5877620a89da53d5" exitCode=0 Oct 30 21:51:45.218112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:45.217691 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" event={"ID":"61634f8d-5849-4ef1-a362-f1d093ba1154","Type":"ContainerDied","Data":"b9a30e457912ce8b7c2909e32784c7116304b8b55a754d7e5877620a89da53d5"} Oct 30 21:51:46.388336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.388310 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:46.492204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.492072 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\") pod \"61634f8d-5849-4ef1-a362-f1d093ba1154\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " Oct 30 21:51:46.493143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.493108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca" pod "61634f8d-5849-4ef1-a362-f1d093ba1154" (UID: "61634f8d-5849-4ef1-a362-f1d093ba1154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:46.493311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.493177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\") pod \"61634f8d-5849-4ef1-a362-f1d093ba1154\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " Oct 30 21:51:46.493311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.493300 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\" (UniqueName: \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\") pod \"61634f8d-5849-4ef1-a362-f1d093ba1154\" (UID: \"61634f8d-5849-4ef1-a362-f1d093ba1154\") " Oct 30 21:51:46.494207 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.494178 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.495572 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.495543 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca" pod "61634f8d-5849-4ef1-a362-f1d093ba1154" (UID: "61634f8d-5849-4ef1-a362-f1d093ba1154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:46.504737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.504707 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8" pod "61634f8d-5849-4ef1-a362-f1d093ba1154" (UID: "61634f8d-5849-4ef1-a362-f1d093ba1154"). InnerVolumeSpecName "kube-api-access-p2wz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:46.595582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.595536 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wz8\" (UniqueName: \"kubernetes.io/projected/61634f8d-5849-4ef1-a362-f1d093ba1154-kube-api-access-p2wz8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.595582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.595577 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61634f8d-5849-4ef1-a362-f1d093ba1154-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.631869 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg"] Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632667 5036 cpu_manager.go:414] "containerMap: removing container" podUID="61634f8d-5849-4ef1-a362-f1d093ba1154" containerName="revoke-keys" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632686 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61634f8d-5849-4ef1-a362-f1d093ba1154" containerName="revoke-keys" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632732 5036 cpu_manager.go:414] "containerMap: removing container" podUID="08153fd3-a1bb-43d5-9643-ba875cf0d511" containerName="revoke-keys" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632739 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08153fd3-a1bb-43d5-9643-ba875cf0d511" containerName="revoke-keys" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632894 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="61634f8d-5849-4ef1-a362-f1d093ba1154" containerName="revoke-keys" Oct 30 21:51:46.634593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.632911 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="08153fd3-a1bb-43d5-9643-ba875cf0d511" containerName="revoke-keys" Oct 30 21:51:46.636980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.636941 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.649286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.649241 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg"] Oct 30 21:51:46.798523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.798439 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzz6\" (UniqueName: \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.798669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.798538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.798717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.798679 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.899668 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.899633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzz6\" (UniqueName: \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.899825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.899684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.899825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.899755 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.900540 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.900401 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.902284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.902248 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.914210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.914182 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzz6\" (UniqueName: \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\") pod \"maas-api-revoke-sub-4b7d151f-t5nvg\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:46.973093 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:46.973064 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:47.232870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:47.232838 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" event={"ID":"61634f8d-5849-4ef1-a362-f1d093ba1154","Type":"ContainerDied","Data":"2ce32827735727596ce52a96dcec88571392a40a2fc0d623c99595d7ba76f10a"} Oct 30 21:51:47.232870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:47.232863 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2" Oct 30 21:51:47.233088 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:47.232892 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce32827735727596ce52a96dcec88571392a40a2fc0d623c99595d7ba76f10a" Oct 30 21:51:47.265525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:47.265504 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg"] Oct 30 21:51:47.267835 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:47.267805 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd40a60b_57a1_4049_9b08_532c103cc575.slice/crio-5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad WatchSource:0}: Error finding container 5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad: Status 404 returned error can't find the container with id 5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad Oct 30 21:51:48.099742 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:51:48.099706 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:51:48.241632 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:48.241597 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd40a60b-57a1-4049-9b08-532c103cc575" containerID="5f661fe8a26d79e8e765a028b0d1adf1726baa3d38cb1746d2919458446763b2" exitCode=0 Oct 30 21:51:48.241820 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:48.241645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" event={"ID":"cd40a60b-57a1-4049-9b08-532c103cc575","Type":"ContainerDied","Data":"5f661fe8a26d79e8e765a028b0d1adf1726baa3d38cb1746d2919458446763b2"} Oct 30 21:51:48.241820 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:48.241689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" event={"ID":"cd40a60b-57a1-4049-9b08-532c103cc575","Type":"ContainerStarted","Data":"5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad"} Oct 30 21:51:49.220138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.220091 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9"] Oct 30 21:51:49.224213 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.224192 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.234550 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.232865 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9"] Oct 30 21:51:49.329092 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.329058 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxqw\" (UniqueName: \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.329320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.329297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.329429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.329393 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.415091 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.415068 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:49.430827 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.430797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxqw\" (UniqueName: \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.430973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.430940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.431043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.431001 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.431637 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.431617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.433824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.433791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.443284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.443245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxqw\" (UniqueName: \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\") pod \"maas-api-revoke-sub-e80ace82-77cr9\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.532510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.532427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\") pod \"cd40a60b-57a1-4049-9b08-532c103cc575\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " Oct 30 21:51:49.532689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.532566 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\") pod \"cd40a60b-57a1-4049-9b08-532c103cc575\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " Oct 30 21:51:49.532689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.532599 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\" (UniqueName: \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\") pod \"cd40a60b-57a1-4049-9b08-532c103cc575\" (UID: \"cd40a60b-57a1-4049-9b08-532c103cc575\") " Oct 30 21:51:49.532898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.532874 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca" pod "cd40a60b-57a1-4049-9b08-532c103cc575" (UID: "cd40a60b-57a1-4049-9b08-532c103cc575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:49.533141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.533123 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:49.535053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.535021 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6" pod "cd40a60b-57a1-4049-9b08-532c103cc575" (UID: "cd40a60b-57a1-4049-9b08-532c103cc575"). InnerVolumeSpecName "kube-api-access-pdzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:49.535189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.535137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca" pod "cd40a60b-57a1-4049-9b08-532c103cc575" (UID: "cd40a60b-57a1-4049-9b08-532c103cc575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:49.540366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.540340 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:49.634737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.634699 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzz6\" (UniqueName: \"kubernetes.io/projected/cd40a60b-57a1-4049-9b08-532c103cc575-kube-api-access-pdzz6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:49.634905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.634742 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd40a60b-57a1-4049-9b08-532c103cc575-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:49.867583 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:49.867560 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9"] Oct 30 21:51:49.869666 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:49.869626 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad559795_c831_4c7a_bdb0_2b7d63b9d881.slice/crio-b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21 WatchSource:0}: Error finding container b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21: Status 404 returned error can't find the container with id b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21 Oct 30 21:51:50.258306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.258238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" event={"ID":"ad559795-c831-4c7a-bdb0-2b7d63b9d881","Type":"ContainerStarted","Data":"ccf87d9b962cecd9ce924c41fb47c4ed5a2a0b76948cb2a41831667142a2f01c"} Oct 30 21:51:50.258770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.258341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" event={"ID":"ad559795-c831-4c7a-bdb0-2b7d63b9d881","Type":"ContainerStarted","Data":"b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21"} Oct 30 21:51:50.259717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.259701 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" Oct 30 21:51:50.259795 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.259754 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg" event={"ID":"cd40a60b-57a1-4049-9b08-532c103cc575","Type":"ContainerDied","Data":"5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad"} Oct 30 21:51:50.260003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.259794 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d44825e368d774c1c4f5fcdcb9220668e224a0ca47d3394c3eed27fecd3dbad" Oct 30 21:51:50.276504 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:50.276448 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" podStartSLOduration=1.276433252 podStartE2EDuration="1.276433252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:50.274594763 +0000 UTC m=+734.224549709" watchObservedRunningTime="2026-10-30 21:51:50.276433252 +0000 UTC m=+734.226388236" Oct 30 21:51:51.268635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:51.268602 5036 generic.go:356] "Generic (PLEG): container finished" podID="ad559795-c831-4c7a-bdb0-2b7d63b9d881" containerID="ccf87d9b962cecd9ce924c41fb47c4ed5a2a0b76948cb2a41831667142a2f01c" exitCode=0 Oct 30 21:51:51.269126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:51.268693 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" event={"ID":"ad559795-c831-4c7a-bdb0-2b7d63b9d881","Type":"ContainerDied","Data":"ccf87d9b962cecd9ce924c41fb47c4ed5a2a0b76948cb2a41831667142a2f01c"} Oct 30 21:51:52.428899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.428873 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:52.569102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.569025 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\") pod \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " Oct 30 21:51:52.569102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.569092 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\") pod \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " Oct 30 21:51:52.569335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.569173 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\" (UniqueName: \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\") pod \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\" (UID: \"ad559795-c831-4c7a-bdb0-2b7d63b9d881\") " Oct 30 21:51:52.569462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.569442 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca" pod "ad559795-c831-4c7a-bdb0-2b7d63b9d881" (UID: "ad559795-c831-4c7a-bdb0-2b7d63b9d881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:52.569652 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.569631 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:52.571398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.571378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca" pod "ad559795-c831-4c7a-bdb0-2b7d63b9d881" (UID: "ad559795-c831-4c7a-bdb0-2b7d63b9d881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:52.571398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.571390 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw" pod "ad559795-c831-4c7a-bdb0-2b7d63b9d881" (UID: "ad559795-c831-4c7a-bdb0-2b7d63b9d881"). InnerVolumeSpecName "kube-api-access-zbxqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:52.671517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.671470 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad559795-c831-4c7a-bdb0-2b7d63b9d881-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:52.671740 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:52.671722 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxqw\" (UniqueName: \"kubernetes.io/projected/ad559795-c831-4c7a-bdb0-2b7d63b9d881-kube-api-access-zbxqw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.285047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:53.285022 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" Oct 30 21:51:53.285239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:53.285046 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9" event={"ID":"ad559795-c831-4c7a-bdb0-2b7d63b9d881","Type":"ContainerDied","Data":"b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21"} Oct 30 21:51:53.285239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:53.285088 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b993f5b20333dd2d78d7b1d78eefd24c0614d9bda2a60a304ddccf2cc72cac21" Oct 30 21:51:55.098597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.098564 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:51:55.099085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099072 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd40a60b-57a1-4049-9b08-532c103cc575" containerName="revoke-keys" Oct 30 21:51:55.099169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099087 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd40a60b-57a1-4049-9b08-532c103cc575" containerName="revoke-keys" Oct 30 21:51:55.099169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099137 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ad559795-c831-4c7a-bdb0-2b7d63b9d881" containerName="revoke-keys" Oct 30 21:51:55.099169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099143 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad559795-c831-4c7a-bdb0-2b7d63b9d881" containerName="revoke-keys" Oct 30 21:51:55.099286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099232 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad559795-c831-4c7a-bdb0-2b7d63b9d881" containerName="revoke-keys" Oct 30 21:51:55.099286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.099241 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd40a60b-57a1-4049-9b08-532c103cc575" containerName="revoke-keys" Oct 30 21:51:55.102666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.102648 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.106576 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.106552 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 21:51:55.109880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.109860 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c709a0b1f5bb\"" Oct 30 21:51:55.109984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.109864 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bb78d58d0bb0\"" Oct 30 21:51:55.110876 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.110629 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tvdlq\"" Oct 30 21:51:55.114880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.114860 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:51:55.203658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.203583 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5zm\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.203658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.203647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.203903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.203731 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.203903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.203791 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.207149 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.207121 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 21:51:55.207653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.207526 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 21:51:55.207808 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.207590 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 21:51:55.304874 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.304836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5zm\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.305114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.304900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.305114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.304967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.305114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.305035 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.306032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.306000 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.306182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.306113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.308494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.308464 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.318703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.318667 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5zm\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\") pod \"payload-processing-external-model-86656fd4cf-hxnjt\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.419429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.419388 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:55.723321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:55.722907 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:51:55.724455 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:55.724411 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb5a0db_2f43_49a3_8db0_4a154d504fc8.slice/crio-33c7ded09cb120c9cb9dea3e2d1ed379db3df3a6bdbcce43e313c255a3d293de WatchSource:0}: Error finding container 33c7ded09cb120c9cb9dea3e2d1ed379db3df3a6bdbcce43e313c255a3d293de: Status 404 returned error can't find the container with id 33c7ded09cb120c9cb9dea3e2d1ed379db3df3a6bdbcce43e313c255a3d293de Oct 30 21:51:56.308770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:56.308732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" event={"ID":"2bb5a0db-2f43-49a3-8db0-4a154d504fc8","Type":"ContainerStarted","Data":"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2"} Oct 30 21:51:56.309282 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:56.308790 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" event={"ID":"2bb5a0db-2f43-49a3-8db0-4a154d504fc8","Type":"ContainerStarted","Data":"33c7ded09cb120c9cb9dea3e2d1ed379db3df3a6bdbcce43e313c255a3d293de"} Oct 30 21:51:56.309282 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:56.308813 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:51:56.336289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:56.335799 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" podStartSLOduration=1.335778947 podStartE2EDuration="1.335778947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:56.327621403 +0000 UTC m=+740.277576351" watchObservedRunningTime="2026-10-30 21:51:56.335778947 +0000 UTC m=+740.285733917" Oct 30 21:51:57.305914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.305883 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs"] Oct 30 21:51:57.310145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.310122 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.335291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.334935 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:57.359290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.346537 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:57.359290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.346620 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:57.369646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.360357 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:51:57.389209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.361934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.389209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.387697 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4j6d\" (UniqueName: \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.389209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.387780 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.389209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.382824 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs"] Oct 30 21:51:57.489165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.489126 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.489348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.489239 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4j6d\" (UniqueName: \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.489348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.489292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.489810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.489782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.491708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.491688 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.501075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.501055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4j6d\" (UniqueName: \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\") pod \"maas-api-revoke-sub-e627f884-q95fs\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:57.691202 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:57.691170 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:51:58.023433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:58.023404 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs"] Oct 30 21:51:58.025086 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:51:58.025055 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679e9c90_c1a7_4e6b_a149_3e5323c01f60.slice/crio-684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5 WatchSource:0}: Error finding container 684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5: Status 404 returned error can't find the container with id 684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5 Oct 30 21:51:58.344170 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:58.344137 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" event={"ID":"679e9c90-c1a7-4e6b-a149-3e5323c01f60","Type":"ContainerStarted","Data":"c2a8f58c9edfbcce6e8d83df496121c0404993d3cb5651277b54161b07b87cf7"} Oct 30 21:51:58.344672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:58.344643 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" event={"ID":"679e9c90-c1a7-4e6b-a149-3e5323c01f60","Type":"ContainerStarted","Data":"684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5"} Oct 30 21:51:58.364231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:58.364178 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" podStartSLOduration=1.3641608170000001 podStartE2EDuration="1.364160817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:58.361240522 +0000 UTC m=+742.311195472" watchObservedRunningTime="2026-10-30 21:51:58.364160817 +0000 UTC m=+742.314115767" Oct 30 21:51:59.353225 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:59.353189 5036 generic.go:356] "Generic (PLEG): container finished" podID="679e9c90-c1a7-4e6b-a149-3e5323c01f60" containerID="c2a8f58c9edfbcce6e8d83df496121c0404993d3cb5651277b54161b07b87cf7" exitCode=0 Oct 30 21:51:59.353645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:51:59.353285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" event={"ID":"679e9c90-c1a7-4e6b-a149-3e5323c01f60","Type":"ContainerDied","Data":"c2a8f58c9edfbcce6e8d83df496121c0404993d3cb5651277b54161b07b87cf7"} Oct 30 21:52:00.575034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.575005 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:52:00.638441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.638407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\") pod \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " Oct 30 21:52:00.638638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.638508 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\") pod \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " Oct 30 21:52:00.638638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.638573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\" (UniqueName: \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\") pod \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\" (UID: \"679e9c90-c1a7-4e6b-a149-3e5323c01f60\") " Oct 30 21:52:00.640959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.640113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca" pod "679e9c90-c1a7-4e6b-a149-3e5323c01f60" (UID: "679e9c90-c1a7-4e6b-a149-3e5323c01f60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:00.658516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.658483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d" pod "679e9c90-c1a7-4e6b-a149-3e5323c01f60" (UID: "679e9c90-c1a7-4e6b-a149-3e5323c01f60"). InnerVolumeSpecName "kube-api-access-j4j6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:00.660141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.660119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca" pod "679e9c90-c1a7-4e6b-a149-3e5323c01f60" (UID: "679e9c90-c1a7-4e6b-a149-3e5323c01f60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:00.742003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.741901 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.742003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.741939 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e9c90-c1a7-4e6b-a149-3e5323c01f60-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.742003 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:00.741960 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4j6d\" (UniqueName: \"kubernetes.io/projected/679e9c90-c1a7-4e6b-a149-3e5323c01f60-kube-api-access-j4j6d\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:01.406951 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:01.406877 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" event={"ID":"679e9c90-c1a7-4e6b-a149-3e5323c01f60","Type":"ContainerDied","Data":"684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5"} Oct 30 21:52:01.406951 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:01.406953 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="684202a221c2a88f2acc60b85ae656fc60619502aef30971a751812b3e7534e5" Oct 30 21:52:01.407240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:01.407032 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs" Oct 30 21:52:04.810211 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.810170 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk"] Oct 30 21:52:04.811034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.811001 5036 cpu_manager.go:414] "containerMap: removing container" podUID="679e9c90-c1a7-4e6b-a149-3e5323c01f60" containerName="revoke-keys" Oct 30 21:52:04.811034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.811031 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679e9c90-c1a7-4e6b-a149-3e5323c01f60" containerName="revoke-keys" Oct 30 21:52:04.811196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.811166 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="679e9c90-c1a7-4e6b-a149-3e5323c01f60" containerName="revoke-keys" Oct 30 21:52:04.815389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.815367 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:04.821965 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.821940 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:52:04.822104 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.822068 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:52:04.822175 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.822139 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:04.822231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.822217 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:52:04.825623 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.825593 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk"] Oct 30 21:52:04.914771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.914741 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:04.914771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.914777 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:04.914982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:04.914877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6rg\" (UniqueName: \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.015750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.015717 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.015750 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.015758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.015991 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.015797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6rg\" (UniqueName: \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.016531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.016501 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.018444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.018420 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.027499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.027475 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6rg\" (UniqueName: \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\") pod \"maas-api-revoke-sub-82f88ce9-9b4dk\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.133414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.133391 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:05.416433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.416399 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk"] Oct 30 21:52:05.418118 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:52:05.418088 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc505b248_de69_468b_83b2_f199f7a8e286.slice/crio-e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f WatchSource:0}: Error finding container e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f: Status 404 returned error can't find the container with id e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f Oct 30 21:52:05.438657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:05.438626 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" event={"ID":"c505b248-de69-468b-83b2-f199f7a8e286","Type":"ContainerStarted","Data":"e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f"} Oct 30 21:52:06.448195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:06.448156 5036 generic.go:356] "Generic (PLEG): container finished" podID="c505b248-de69-468b-83b2-f199f7a8e286" containerID="12deaa2cb6c595c5a170daa8aeabf6979dbd0245bfc668dac6cb0bc8bd970c83" exitCode=0 Oct 30 21:52:06.448600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:06.448239 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" event={"ID":"c505b248-de69-468b-83b2-f199f7a8e286","Type":"ContainerDied","Data":"12deaa2cb6c595c5a170daa8aeabf6979dbd0245bfc668dac6cb0bc8bd970c83"} Oct 30 21:52:07.050580 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.050549 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 21:52:07.054567 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.054546 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.058017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.057993 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ceb36a-openshift-default-dockercfg-vtb7n\"" Oct 30 21:52:07.082963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.082931 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 21:52:07.140306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140258 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140617 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg6c\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140706 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140739 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.140780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.140771 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241319 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241390 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg6c\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241442 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241484 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241560 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241649 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.241938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.241867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.242188 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.242155 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.242188 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.242179 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.242411 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.242215 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.244304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.244258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.244477 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.244455 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.254344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.254322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.254921 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.254897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg6c\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.338097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.338018 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:52:07.374685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.374657 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:07.624453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.624425 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:07.674242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.674220 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 21:52:07.675720 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:52:07.675694 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204de64f_65a6_4094_baa1_977345403ba9.slice/crio-de6cebe147de016dbf311c818d4b9725948a3f83fe0c8ce40d6eda6b0f1ae79f WatchSource:0}: Error finding container de6cebe147de016dbf311c818d4b9725948a3f83fe0c8ce40d6eda6b0f1ae79f: Status 404 returned error can't find the container with id de6cebe147de016dbf311c818d4b9725948a3f83fe0c8ce40d6eda6b0f1ae79f Oct 30 21:52:07.678565 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.678526 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:07.678650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.678612 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:07.678688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.678658 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:07.747248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.747216 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\" (UniqueName: \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\") pod \"c505b248-de69-468b-83b2-f199f7a8e286\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " Oct 30 21:52:07.747488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.747465 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\") pod \"c505b248-de69-468b-83b2-f199f7a8e286\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " Oct 30 21:52:07.747593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.747513 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\") pod \"c505b248-de69-468b-83b2-f199f7a8e286\" (UID: \"c505b248-de69-468b-83b2-f199f7a8e286\") " Oct 30 21:52:07.748034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.747989 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca" pod "c505b248-de69-468b-83b2-f199f7a8e286" (UID: "c505b248-de69-468b-83b2-f199f7a8e286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:07.749691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.749654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca" pod "c505b248-de69-468b-83b2-f199f7a8e286" (UID: "c505b248-de69-468b-83b2-f199f7a8e286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:07.750117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.750085 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg" pod "c505b248-de69-468b-83b2-f199f7a8e286" (UID: "c505b248-de69-468b-83b2-f199f7a8e286"). InnerVolumeSpecName "kube-api-access-tw6rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:07.849569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.849528 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6rg\" (UniqueName: \"kubernetes.io/projected/c505b248-de69-468b-83b2-f199f7a8e286-kube-api-access-tw6rg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:07.849569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.849561 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:07.849569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:07.849577 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c505b248-de69-468b-83b2-f199f7a8e286-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:08.467938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.467837 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" Oct 30 21:52:08.467938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.467879 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk" event={"ID":"c505b248-de69-468b-83b2-f199f7a8e286","Type":"ContainerDied","Data":"e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f"} Oct 30 21:52:08.467938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.467937 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7b93247f1aba7ba1ecd2a5235565337b216552a16646ecb9a6e3a8dd5f3920f" Oct 30 21:52:08.469774 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.469746 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" event={"ID":"204de64f-65a6-4094-baa1-977345403ba9","Type":"ContainerStarted","Data":"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301"} Oct 30 21:52:08.470010 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.469792 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" event={"ID":"204de64f-65a6-4094-baa1-977345403ba9","Type":"ContainerStarted","Data":"de6cebe147de016dbf311c818d4b9725948a3f83fe0c8ce40d6eda6b0f1ae79f"} Oct 30 21:52:08.492020 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:08.491966 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podStartSLOduration=1.491949803 podStartE2EDuration="1.491949803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:08.491385917 +0000 UTC m=+752.441340895" watchObservedRunningTime="2026-10-30 21:52:08.491949803 +0000 UTC m=+752.441904749" Oct 30 21:52:09.150290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.149330 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 21:52:09.174290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.173417 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c505b248-de69-468b-83b2-f199f7a8e286" containerName="revoke-keys" Oct 30 21:52:09.174290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.173449 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c505b248-de69-468b-83b2-f199f7a8e286" containerName="revoke-keys" Oct 30 21:52:09.174290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.173712 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c505b248-de69-468b-83b2-f199f7a8e286" containerName="revoke-keys" Oct 30 21:52:09.200654 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.200625 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.206302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.206255 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-939a16-openshift-default-dockercfg-6sr4l\"" Oct 30 21:52:09.215321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.210993 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.277909 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278034 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4km\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278131 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278158 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278217 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278320 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278366 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.278797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.278409 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.375699 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.375661 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:09.377288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.377237 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:09.377410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.377303 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:09.379821 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.379799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.379931 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.379883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4km\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.379931 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.379922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.379968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.379991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380096 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380315 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380448 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380390 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380517 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380544 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380687 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.380816 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.381010 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.380990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.382518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.382496 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.382658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.382641 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.391765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.391742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4km\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.391865 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.391775 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.554809 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.554713 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:09.872932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.872907 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 21:52:09.874694 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:52:09.874656 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47182a02_578a_4d7b_85f9_5bd0352a0f9e.slice/crio-b446fddc6a656990c455f041e339fa69aabe3cfffd1306d6ac5ff668de85fd9a WatchSource:0}: Error finding container b446fddc6a656990c455f041e339fa69aabe3cfffd1306d6ac5ff668de85fd9a: Status 404 returned error can't find the container with id b446fddc6a656990c455f041e339fa69aabe3cfffd1306d6ac5ff668de85fd9a Oct 30 21:52:09.877370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.877330 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:09.877472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.877408 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:09.877472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:09.877440 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:10.375105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.375068 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:10.375520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.375134 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:10.495291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.486429 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 21:52:10.495704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.495675 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.503063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.503025 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0f26c5-openshift-default-dockercfg-tmpf5\"" Oct 30 21:52:10.504684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.504588 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" event={"ID":"47182a02-578a-4d7b-85f9-5bd0352a0f9e","Type":"ContainerStarted","Data":"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a"} Oct 30 21:52:10.504684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.504634 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" event={"ID":"47182a02-578a-4d7b-85f9-5bd0352a0f9e","Type":"ContainerStarted","Data":"b446fddc6a656990c455f041e339fa69aabe3cfffd1306d6ac5ff668de85fd9a"} Oct 30 21:52:10.506529 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.506039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 21:52:10.582325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.574921 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:10.582325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.575051 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:10.582325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.575093 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:10.597488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.597458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.597734 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.597718 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.597945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.597933 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598134 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598219 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598474 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598448 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j774\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598540 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598497 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.598575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.598544 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699390 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699434 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699461 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699521 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j774\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699543 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699573 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699610 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.699837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.700002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.699980 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.700463 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.700439 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.700594 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.700506 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.700789 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.700768 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.700855 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.700803 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.703156 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.703123 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.703452 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.703429 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.752288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.749835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.758290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.753951 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j774\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:10.839290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:10.839063 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:11.269982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.257063 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podStartSLOduration=2.257039237 podStartE2EDuration="2.257039237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:10.572966877 +0000 UTC m=+754.522921844" watchObservedRunningTime="2026-10-30 21:52:11.257039237 +0000 UTC m=+755.206994184" Oct 30 21:52:11.269982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.269777 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 21:52:11.274845 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:52:11.274804 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59612a38_c52a_48b7_9209_2e36fbc8007c.slice/crio-44feefa8a0dd919e1a02af6d00fd56abc94f2554e2369116b83d25081f517e26 WatchSource:0}: Error finding container 44feefa8a0dd919e1a02af6d00fd56abc94f2554e2369116b83d25081f517e26: Status 404 returned error can't find the container with id 44feefa8a0dd919e1a02af6d00fd56abc94f2554e2369116b83d25081f517e26 Oct 30 21:52:11.277146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.277106 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:11.277314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.277179 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:11.277314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.277210 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:11.375562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.375535 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:11.375967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.375584 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:11.525346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.524840 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" event={"ID":"59612a38-c52a-48b7-9209-2e36fbc8007c","Type":"ContainerStarted","Data":"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0"} Oct 30 21:52:11.525346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.524901 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" event={"ID":"59612a38-c52a-48b7-9209-2e36fbc8007c","Type":"ContainerStarted","Data":"44feefa8a0dd919e1a02af6d00fd56abc94f2554e2369116b83d25081f517e26"} Oct 30 21:52:11.559287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.559183 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:11.559287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.559245 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:11.842527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:11.842421 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:12.380293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.379700 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:12.380293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.379764 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:12.607309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.606821 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:12.607309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.606876 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:12.841293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.841121 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:12.841293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:12.841184 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:13.376288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.375696 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:13.376288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.375775 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:13.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.558431 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:13.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.558497 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:13.839756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.839717 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:13.840043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:13.840017 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:14.378767 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.378725 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:52:14.379048 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.379015 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:52:14.555760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.555720 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:14.556083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.556042 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:14.839800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.839765 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:14.840460 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:14.840409 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:15.059293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.058644 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podStartSLOduration=5.058621823 podStartE2EDuration="5.058621823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:11.55872643 +0000 UTC m=+755.508681377" watchObservedRunningTime="2026-10-30 21:52:15.058621823 +0000 UTC m=+759.008576769" Oct 30 21:52:15.063289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.059717 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf"] Oct 30 21:52:15.072810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.072781 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.091336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.087240 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf"] Oct 30 21:52:15.097480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.091601 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:15.097653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.091669 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:52:15.097781 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.091682 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:52:15.097886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.091723 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:52:15.102624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.102593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7r55\" (UniqueName: \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.102744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.102713 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.103124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.103104 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.204644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.204603 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.204834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.204703 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.204950 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.204931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7r55\" (UniqueName: \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.206484 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.206444 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.208082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.208057 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.244071 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.243990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7r55\" (UniqueName: \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\") pod \"maas-api-revoke-sub-a52f6e27-znvdf\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.422528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.422498 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:15.561100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.560652 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:15.561100 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.560713 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:15.842472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.840741 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:15.842472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:15.840805 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:16.194827 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.194790 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:16.195321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.195296 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:16.196371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.196356 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 21:52:16.396026 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.395992 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf"] Oct 30 21:52:16.557696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.557662 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:52:16.557997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.557963 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:52:16.781495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.781457 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" event={"ID":"2d658eb4-6d6d-4be3-a388-b11caa796ca9","Type":"ContainerStarted","Data":"de44ba3880f3c1a207c2710f926474f3a620fde1d365a9e4b4e82e7be880dc34"} Oct 30 21:52:16.851291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.850732 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:16.851291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:16.850796 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:17.561404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.561375 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:17.562080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.562059 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:17.562761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.562741 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 21:52:17.792132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.792093 5036 generic.go:356] "Generic (PLEG): container finished" podID="2d658eb4-6d6d-4be3-a388-b11caa796ca9" containerID="8d9549edabfe93d4e90fd65d61aacc61f9948ea8fdacc066116d4ca6fa9846de" exitCode=0 Oct 30 21:52:17.792357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.792174 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" event={"ID":"2d658eb4-6d6d-4be3-a388-b11caa796ca9","Type":"ContainerDied","Data":"8d9549edabfe93d4e90fd65d61aacc61f9948ea8fdacc066116d4ca6fa9846de"} Oct 30 21:52:17.839639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.839555 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:52:17.839787 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:17.839629 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:52:18.139285 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:52:18.136797 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:52:18.876933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:18.871571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:18.876933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:18.874553 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:18.882894 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:18.882831 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 21:52:19.295450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.295429 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:19.449713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.449670 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\" (UniqueName: \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\") pod \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " Oct 30 21:52:19.474289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.450082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\") pod \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " Oct 30 21:52:19.474289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.450332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\") pod \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\" (UID: \"2d658eb4-6d6d-4be3-a388-b11caa796ca9\") " Oct 30 21:52:19.474289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.468676 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca" pod "2d658eb4-6d6d-4be3-a388-b11caa796ca9" (UID: "2d658eb4-6d6d-4be3-a388-b11caa796ca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:19.474289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.468761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca" pod "2d658eb4-6d6d-4be3-a388-b11caa796ca9" (UID: "2d658eb4-6d6d-4be3-a388-b11caa796ca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:19.499537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.499504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55" pod "2d658eb4-6d6d-4be3-a388-b11caa796ca9" (UID: "2d658eb4-6d6d-4be3-a388-b11caa796ca9"). InnerVolumeSpecName "kube-api-access-t7r55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:19.576290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.576113 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:19.576290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.576166 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7r55\" (UniqueName: \"kubernetes.io/projected/2d658eb4-6d6d-4be3-a388-b11caa796ca9-kube-api-access-t7r55\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:19.576290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.576186 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d658eb4-6d6d-4be3-a388-b11caa796ca9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:19.836664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.836577 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" Oct 30 21:52:19.848290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.836871 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf" event={"ID":"2d658eb4-6d6d-4be3-a388-b11caa796ca9","Type":"ContainerDied","Data":"de44ba3880f3c1a207c2710f926474f3a620fde1d365a9e4b4e82e7be880dc34"} Oct 30 21:52:19.848290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:19.840374 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de44ba3880f3c1a207c2710f926474f3a620fde1d365a9e4b4e82e7be880dc34" Oct 30 21:52:48.101516 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:52:48.101475 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:52:55.909972 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:52:55.909938 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:53:00.911076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:00.911033 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" podUID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" containerName="payload-processing" containerID="cri-o://aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2" gracePeriod=25 Oct 30 21:53:01.188212 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.188179 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:53:01.301869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.301839 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\") pod \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " Oct 30 21:53:01.302062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.301900 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\") pod \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " Oct 30 21:53:01.302062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302020 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\") pod \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " Oct 30 21:53:01.302221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\") pod \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\" (UID: \"2bb5a0db-2f43-49a3-8db0-4a154d504fc8\") " Oct 30 21:53:01.302402 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302369 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume" pod "2bb5a0db-2f43-49a3-8db0-4a154d504fc8" (UID: "2bb5a0db-2f43-49a3-8db0-4a154d504fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:01.302564 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302541 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume" pod "2bb5a0db-2f43-49a3-8db0-4a154d504fc8" (UID: "2bb5a0db-2f43-49a3-8db0-4a154d504fc8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:01.302754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302734 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:01.302754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.302753 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:01.304480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.304458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm" pod "2bb5a0db-2f43-49a3-8db0-4a154d504fc8" (UID: "2bb5a0db-2f43-49a3-8db0-4a154d504fc8"). InnerVolumeSpecName "kube-api-access-7v5zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:01.304480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.304475 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials" pod "2bb5a0db-2f43-49a3-8db0-4a154d504fc8" (UID: "2bb5a0db-2f43-49a3-8db0-4a154d504fc8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:01.403610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.403572 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5zm\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-kube-api-access-7v5zm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:01.403610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.403604 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb5a0db-2f43-49a3-8db0-4a154d504fc8-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:01.443511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.443411 5036 generic.go:356] "Generic (PLEG): container finished" podID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" containerID="aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2" exitCode=0 Oct 30 21:53:01.443511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.443466 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" Oct 30 21:53:01.443798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.443510 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" event={"ID":"2bb5a0db-2f43-49a3-8db0-4a154d504fc8","Type":"ContainerDied","Data":"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2"} Oct 30 21:53:01.443798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.443570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt" event={"ID":"2bb5a0db-2f43-49a3-8db0-4a154d504fc8","Type":"ContainerDied","Data":"33c7ded09cb120c9cb9dea3e2d1ed379db3df3a6bdbcce43e313c255a3d293de"} Oct 30 21:53:01.443798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.443595 5036 scope.go:122] "RemoveContainer" containerID="aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2" Oct 30 21:53:01.454737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.454717 5036 scope.go:122] "RemoveContainer" containerID="aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2" Oct 30 21:53:01.455024 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:53:01.455007 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2\": container with ID starting with aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2 not found: ID does not exist" containerID="aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2" Oct 30 21:53:01.455105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.455032 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2"} err="failed to get container status \"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2\": rpc error: code = NotFound desc = could not find container \"aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2\": container with ID starting with aa5aeadb1de28374e38234e0adda9c51af481b44f7de7f4a5a92ef845a4c6aa2 not found: ID does not exist" Oct 30 21:53:01.468312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.468248 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:53:01.474260 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:01.474236 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86656fd4cf-hxnjt"] Oct 30 21:53:02.188932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:02.188899 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" path="/var/lib/kubelet/pods/2bb5a0db-2f43-49a3-8db0-4a154d504fc8/volumes" Oct 30 21:53:07.169844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.169811 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 21:53:07.171051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171016 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2d658eb4-6d6d-4be3-a388-b11caa796ca9" containerName="revoke-keys" Oct 30 21:53:07.171051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171054 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d658eb4-6d6d-4be3-a388-b11caa796ca9" containerName="revoke-keys" Oct 30 21:53:07.171258 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171158 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" containerName="payload-processing" Oct 30 21:53:07.171258 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171173 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" containerName="payload-processing" Oct 30 21:53:07.171503 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171481 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb5a0db-2f43-49a3-8db0-4a154d504fc8" containerName="payload-processing" Oct 30 21:53:07.171567 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.171517 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d658eb4-6d6d-4be3-a388-b11caa796ca9" containerName="revoke-keys" Oct 30 21:53:07.183696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.183667 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.200502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.200445 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 21:53:07.275849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.275807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.276524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.276494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.276729 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.276708 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.277084 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.277062 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.277225 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.277209 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.277471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.277450 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.277961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.277876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcqz\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.277961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.277936 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.278138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.278053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378716 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378778 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378813 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378843 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.378930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378960 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcqz\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.378997 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.379054 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.379288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.379517 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.379568 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.379973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.379946 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.380072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.380035 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.381687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.381666 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.382287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.382249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.391562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.391537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcqz\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.392117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.392074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.562375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.562295 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:07.717292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.716577 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj"] Oct 30 21:53:07.723894 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.721973 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.733656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.731386 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:07.733656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.731434 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:07.733656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.731506 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:07.733656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.731911 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:07.735199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.735176 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj"] Oct 30 21:53:07.787882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.787855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.788042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.787893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchjv\" (UniqueName: \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.788042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.787982 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.872869 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:07.872838 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373af11a_0efb_4536_abba_e9246f5aedb2.slice/crio-52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd WatchSource:0}: Error finding container 52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd: Status 404 returned error can't find the container with id 52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd Oct 30 21:53:07.875950 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.875906 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:07.876057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.875996 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:07.876057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.876037 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:07.878798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.878777 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 21:53:07.889484 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.889458 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.889643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.889626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.889687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.889657 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchjv\" (UniqueName: \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.890336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.890312 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.892034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.892012 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:07.906858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:07.906840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchjv\" (UniqueName: \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\") pod \"maas-api-revoke-sub-f3dfc762-nzgkj\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:08.069129 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.069089 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:08.382646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.382282 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj"] Oct 30 21:53:08.390800 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:08.390756 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74f9f78_2df3_4358_9ec7_66cfd3f05287.slice/crio-fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db WatchSource:0}: Error finding container fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db: Status 404 returned error can't find the container with id fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db Oct 30 21:53:08.502169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.502127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" event={"ID":"d74f9f78-2df3-4358-9ec7-66cfd3f05287","Type":"ContainerStarted","Data":"fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db"} Oct 30 21:53:08.504163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.504137 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" event={"ID":"373af11a-0efb-4536-abba-e9246f5aedb2","Type":"ContainerStarted","Data":"a43541d9c78265932433ff9c48bb8d5ddeb10b89d088603f67a256888f88ccd1"} Oct 30 21:53:08.504332 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.504185 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" event={"ID":"373af11a-0efb-4536-abba-e9246f5aedb2","Type":"ContainerStarted","Data":"52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd"} Oct 30 21:53:08.530617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.530564 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podStartSLOduration=1.530546888 podStartE2EDuration="1.530546888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:08.527149708 +0000 UTC m=+812.477104656" watchObservedRunningTime="2026-10-30 21:53:08.530546888 +0000 UTC m=+812.480501833" Oct 30 21:53:08.562525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.562496 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:08.564149 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.564122 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:08.564315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:08.564176 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:09.249057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.249018 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 21:53:09.254354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.254330 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.270173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.270144 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 21:53:09.309169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309131 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309367 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309367 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309286 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309367 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309432 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309517 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt7z\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309569 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.309674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.309637 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.410877 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.410962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.410986 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411036 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411088 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411166 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt7z\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411192 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.414292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.411314 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.415210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.414675 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.415210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.414944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.415351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.415209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.415899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.415874 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.416882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.416858 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.446361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.436847 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.446361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.441929 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt7z\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.446361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.443239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.446855 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.446830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.534810 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:53:09.534696 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74f9f78_2df3_4358_9ec7_66cfd3f05287.slice/crio-conmon-437d53e358f79812c551cab8445a59c9597b600959c2ed087d10a426bdd61521.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:53:09.541290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.538651 5036 generic.go:356] "Generic (PLEG): container finished" podID="d74f9f78-2df3-4358-9ec7-66cfd3f05287" containerID="437d53e358f79812c551cab8445a59c9597b600959c2ed087d10a426bdd61521" exitCode=0 Oct 30 21:53:09.541290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.539933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" event={"ID":"d74f9f78-2df3-4358-9ec7-66cfd3f05287","Type":"ContainerDied","Data":"437d53e358f79812c551cab8445a59c9597b600959c2ed087d10a426bdd61521"} Oct 30 21:53:09.563382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.563352 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:09.563540 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.563400 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:09.574932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.574904 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:09.943492 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:09.943458 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e44091_8681_421c_906f_b145c25aa6a3.slice/crio-8350c9eb27d2ee2d55c929e242f7464f41f832efa448c3ae45d0c1169db2ca3a WatchSource:0}: Error finding container 8350c9eb27d2ee2d55c929e242f7464f41f832efa448c3ae45d0c1169db2ca3a: Status 404 returned error can't find the container with id 8350c9eb27d2ee2d55c929e242f7464f41f832efa448c3ae45d0c1169db2ca3a Oct 30 21:53:09.944139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.944078 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 21:53:09.946108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.946077 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:09.946189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.946140 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:09.946189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:09.946174 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:10.562296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.562232 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" event={"ID":"33e44091-8681-421c-906f-b145c25aa6a3","Type":"ContainerStarted","Data":"373783f7c42b3ce58452dc9e0262220fa76871b5957ba6a75680593aa643063c"} Oct 30 21:53:10.563530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.562820 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" event={"ID":"33e44091-8681-421c-906f-b145c25aa6a3","Type":"ContainerStarted","Data":"8350c9eb27d2ee2d55c929e242f7464f41f832efa448c3ae45d0c1169db2ca3a"} Oct 30 21:53:10.570287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.568400 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:10.570287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.568459 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:10.583603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.581025 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:10.600330 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.599891 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podStartSLOduration=1.599869074 podStartE2EDuration="1.599869074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:10.597066933 +0000 UTC m=+814.547021881" watchObservedRunningTime="2026-10-30 21:53:10.599869074 +0000 UTC m=+814.549824021" Oct 30 21:53:10.957291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:10.956903 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:11.094309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.081607 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\") pod \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " Oct 30 21:53:11.094309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.081702 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\") pod \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " Oct 30 21:53:11.094309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.081744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\" (UniqueName: \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\") pod \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\" (UID: \"d74f9f78-2df3-4358-9ec7-66cfd3f05287\") " Oct 30 21:53:11.094309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.083161 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca" pod "d74f9f78-2df3-4358-9ec7-66cfd3f05287" (UID: "d74f9f78-2df3-4358-9ec7-66cfd3f05287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:11.097348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.096572 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca" pod "d74f9f78-2df3-4358-9ec7-66cfd3f05287" (UID: "d74f9f78-2df3-4358-9ec7-66cfd3f05287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:11.097348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.096826 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv" pod "d74f9f78-2df3-4358-9ec7-66cfd3f05287" (UID: "d74f9f78-2df3-4358-9ec7-66cfd3f05287"). InnerVolumeSpecName "kube-api-access-tchjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:11.183087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.182994 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.183087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.183034 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74f9f78-2df3-4358-9ec7-66cfd3f05287-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.183087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.183050 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchjv\" (UniqueName: \"kubernetes.io/projected/d74f9f78-2df3-4358-9ec7-66cfd3f05287-kube-api-access-tchjv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.563366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.562798 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:11.563366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.562875 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:11.577137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.577043 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:11.577437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.577408 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:11.578784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.578769 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" Oct 30 21:53:11.578975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.578818 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj" event={"ID":"d74f9f78-2df3-4358-9ec7-66cfd3f05287","Type":"ContainerDied","Data":"fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db"} Oct 30 21:53:11.579030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:11.579011 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbc0aeb69571edd335a0700da5e26743aa4c39681ed3cbe6629bcd14251e03db" Oct 30 21:53:12.568669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:12.568629 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:12.569322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:12.569283 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:12.576647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:12.576618 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:12.576846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:12.576820 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:13.566681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:13.566623 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:13.566871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:13.566704 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:13.576207 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:13.576176 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:13.576621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:13.576225 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:14.332625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.332577 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t"] Oct 30 21:53:14.333747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.333720 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d74f9f78-2df3-4358-9ec7-66cfd3f05287" containerName="revoke-keys" Oct 30 21:53:14.333909 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.333895 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74f9f78-2df3-4358-9ec7-66cfd3f05287" containerName="revoke-keys" Oct 30 21:53:14.334163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.334150 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74f9f78-2df3-4358-9ec7-66cfd3f05287" containerName="revoke-keys" Oct 30 21:53:14.355732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.355704 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t"] Oct 30 21:53:14.356008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.355991 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.364395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.361196 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:14.364395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.361403 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:14.364395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.361445 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:14.364395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.361513 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:14.477612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.477576 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxt5\" (UniqueName: \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.477760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.477706 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.477828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.477758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.562832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.562795 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:53:14.563104 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.563077 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:53:14.575370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.575337 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:14.575506 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.575404 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:14.578549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.578513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.579004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.578589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.579004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.578765 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxt5\" (UniqueName: \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.580124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.580094 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.583591 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.583539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.591856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.591830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxt5\" (UniqueName: \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\") pod \"maas-api-revoke-sub-440b92b3-5vt4t\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:14.679858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:14.679822 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:15.002245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.002219 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t"] Oct 30 21:53:15.004229 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:15.004204 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeeeb53_52e6_4d1a_be28_45e4690cec3f.slice/crio-bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb WatchSource:0}: Error finding container bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb: Status 404 returned error can't find the container with id bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb Oct 30 21:53:15.577241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.576386 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:15.577241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.576444 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:15.582756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.582724 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:15.584557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.584504 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:15.586045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.586009 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 21:53:15.652314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.645150 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" event={"ID":"cbeeeb53-52e6-4d1a-be28-45e4690cec3f","Type":"ContainerStarted","Data":"1c29f033b9cba01a55a2e746b6b48bafbd6b83ed8f682aedcd54c51cc8bd7ecd"} Oct 30 21:53:15.652314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.645208 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" event={"ID":"cbeeeb53-52e6-4d1a-be28-45e4690cec3f","Type":"ContainerStarted","Data":"bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb"} Oct 30 21:53:15.681169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:15.681113 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" podStartSLOduration=1.681093654 podStartE2EDuration="1.681093654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:15.676043838 +0000 UTC m=+819.625998785" watchObservedRunningTime="2026-10-30 21:53:15.681093654 +0000 UTC m=+819.631048602" Oct 30 21:53:16.575307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:16.575258 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:53:16.575496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:16.575338 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:53:16.655518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:16.655436 5036 generic.go:356] "Generic (PLEG): container finished" podID="cbeeeb53-52e6-4d1a-be28-45e4690cec3f" containerID="1c29f033b9cba01a55a2e746b6b48bafbd6b83ed8f682aedcd54c51cc8bd7ecd" exitCode=0 Oct 30 21:53:16.656017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:16.655530 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" event={"ID":"cbeeeb53-52e6-4d1a-be28-45e4690cec3f","Type":"ContainerDied","Data":"1c29f033b9cba01a55a2e746b6b48bafbd6b83ed8f682aedcd54c51cc8bd7ecd"} Oct 30 21:53:17.607289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:17.601687 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:17.607289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:17.603210 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:17.607289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:17.606313 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 21:53:18.046188 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.046160 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:18.181016 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:53:18.180621 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:53:18.193050 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.191678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\") pod \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " Oct 30 21:53:18.193050 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.191779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\" (UniqueName: \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\") pod \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " Oct 30 21:53:18.193050 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.192089 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\") pod \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\" (UID: \"cbeeeb53-52e6-4d1a-be28-45e4690cec3f\") " Oct 30 21:53:18.193759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.193504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca" pod "cbeeeb53-52e6-4d1a-be28-45e4690cec3f" (UID: "cbeeeb53-52e6-4d1a-be28-45e4690cec3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:18.198393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.198355 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5" pod "cbeeeb53-52e6-4d1a-be28-45e4690cec3f" (UID: "cbeeeb53-52e6-4d1a-be28-45e4690cec3f"). InnerVolumeSpecName "kube-api-access-txxt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:18.227042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.225654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca" pod "cbeeeb53-52e6-4d1a-be28-45e4690cec3f" (UID: "cbeeeb53-52e6-4d1a-be28-45e4690cec3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:18.296370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.296279 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.296370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.296318 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.296370 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.296336 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxt5\" (UniqueName: \"kubernetes.io/projected/cbeeeb53-52e6-4d1a-be28-45e4690cec3f-kube-api-access-txxt5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.683782 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.683748 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" Oct 30 21:53:18.683988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.683785 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t" event={"ID":"cbeeeb53-52e6-4d1a-be28-45e4690cec3f","Type":"ContainerDied","Data":"bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb"} Oct 30 21:53:18.683988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:18.683834 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad4b8a9f1d37d6f8bfd37914b72b7dbfa3811c85ede28b48478e0a464dd46bb" Oct 30 21:53:19.837805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.837772 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf"] Oct 30 21:53:19.838439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.838422 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cbeeeb53-52e6-4d1a-be28-45e4690cec3f" containerName="revoke-keys" Oct 30 21:53:19.838510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.838440 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbeeeb53-52e6-4d1a-be28-45e4690cec3f" containerName="revoke-keys" Oct 30 21:53:19.838560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.838549 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbeeeb53-52e6-4d1a-be28-45e4690cec3f" containerName="revoke-keys" Oct 30 21:53:19.842325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.842306 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:19.847125 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.847103 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:19.847384 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.847186 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:19.847706 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.847691 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:19.847800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.847732 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:19.853214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.853193 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf"] Oct 30 21:53:19.920688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.920661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:19.920841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.920735 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxhz\" (UniqueName: \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:19.920905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:19.920885 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.021938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.021905 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.022122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.021999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.022122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.022081 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxhz\" (UniqueName: \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.022755 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.022729 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.024599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.024580 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.043804 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.043783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxhz\" (UniqueName: \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\") pod \"maas-api-revoke-sub-35678ea2-dvzgf\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.162910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.162874 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:20.466986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.466956 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf"] Oct 30 21:53:20.467305 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:20.467245 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce2f42c_0d56_4e61_b04f_01a993b10388.slice/crio-9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b WatchSource:0}: Error finding container 9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b: Status 404 returned error can't find the container with id 9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b Oct 30 21:53:20.621462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.621431 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m"] Oct 30 21:53:20.645534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.645508 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m"] Oct 30 21:53:20.645879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.645861 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.700742 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.700710 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" event={"ID":"4ce2f42c-0d56-4e61-b04f-01a993b10388","Type":"ContainerStarted","Data":"9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b"} Oct 30 21:53:20.731971 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.731892 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.731971 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.731944 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zf6x\" (UniqueName: \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.732209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.732184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.833461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.833433 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.833556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.833547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.833603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.833567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zf6x\" (UniqueName: \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.834040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.834014 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.835771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.835750 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.845564 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.845544 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zf6x\" (UniqueName: \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\") pod \"maas-api-revoke-sub-6efe13b0-b5q9m\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:20.962935 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:20.962900 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:21.251549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.251500 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m"] Oct 30 21:53:21.252178 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:21.252146 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5661e199_85a5_4a15_9ad5_720233255d95.slice/crio-6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933 WatchSource:0}: Error finding container 6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933: Status 404 returned error can't find the container with id 6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933 Oct 30 21:53:21.710231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.710192 5036 generic.go:356] "Generic (PLEG): container finished" podID="4ce2f42c-0d56-4e61-b04f-01a993b10388" containerID="94bad482edcb626971532a1eafb95c24d3589ef863c315a15f020b2e04c47880" exitCode=0 Oct 30 21:53:21.710461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.710240 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" event={"ID":"4ce2f42c-0d56-4e61-b04f-01a993b10388","Type":"ContainerDied","Data":"94bad482edcb626971532a1eafb95c24d3589ef863c315a15f020b2e04c47880"} Oct 30 21:53:21.712549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.712514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" event={"ID":"5661e199-85a5-4a15-9ad5-720233255d95","Type":"ContainerStarted","Data":"28a1710b51ab07dc5e803171c820e4f8831ce4ebd380912b2992ce6f9c9a8ba0"} Oct 30 21:53:21.712664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.712571 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" event={"ID":"5661e199-85a5-4a15-9ad5-720233255d95","Type":"ContainerStarted","Data":"6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933"} Oct 30 21:53:21.747926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.747865 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" podStartSLOduration=1.747849365 podStartE2EDuration="1.747849365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:21.746417636 +0000 UTC m=+825.696372585" watchObservedRunningTime="2026-10-30 21:53:21.747849365 +0000 UTC m=+825.697804310" Oct 30 21:53:21.890017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.889984 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 21:53:21.894315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.894295 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.915530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.915501 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 21:53:21.948239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948209 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948415 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948257 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948415 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948535 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948572 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbf5\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:21.948856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:21.948839 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049461 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049505 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049587 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049747 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.049908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.049800 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbf5\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.050260 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.050236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.050575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.050552 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.050630 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.050598 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.050855 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.050834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.051005 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.050983 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.054538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.054513 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.054843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.054823 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.066001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.065980 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbf5\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.070443 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.070423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.209771 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.209700 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:22.520463 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:22.520429 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce65d24c_4d03_4c1d_a614_760edd8f1c42.slice/crio-4758015d96738ba40d2cf0218a53ebc33bd32bb53993fac189915f411d7faab4 WatchSource:0}: Error finding container 4758015d96738ba40d2cf0218a53ebc33bd32bb53993fac189915f411d7faab4: Status 404 returned error can't find the container with id 4758015d96738ba40d2cf0218a53ebc33bd32bb53993fac189915f411d7faab4 Oct 30 21:53:22.520605 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.520538 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 21:53:22.523797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.523765 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:22.523908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.523825 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:22.523908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.523856 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:22.746290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.746062 5036 generic.go:356] "Generic (PLEG): container finished" podID="5661e199-85a5-4a15-9ad5-720233255d95" containerID="28a1710b51ab07dc5e803171c820e4f8831ce4ebd380912b2992ce6f9c9a8ba0" exitCode=0 Oct 30 21:53:22.746290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.746179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" event={"ID":"5661e199-85a5-4a15-9ad5-720233255d95","Type":"ContainerDied","Data":"28a1710b51ab07dc5e803171c820e4f8831ce4ebd380912b2992ce6f9c9a8ba0"} Oct 30 21:53:22.754292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.749464 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" event={"ID":"ce65d24c-4d03-4c1d-a614-760edd8f1c42","Type":"ContainerStarted","Data":"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76"} Oct 30 21:53:22.754292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.749514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" event={"ID":"ce65d24c-4d03-4c1d-a614-760edd8f1c42","Type":"ContainerStarted","Data":"4758015d96738ba40d2cf0218a53ebc33bd32bb53993fac189915f411d7faab4"} Oct 30 21:53:22.802063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.801930 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podStartSLOduration=1.801907022 podStartE2EDuration="1.801907022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:22.795559985 +0000 UTC m=+826.745514943" watchObservedRunningTime="2026-10-30 21:53:22.801907022 +0000 UTC m=+826.751861994" Oct 30 21:53:22.914683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.914654 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:22.965004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.964962 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\" (UniqueName: \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\") pod \"4ce2f42c-0d56-4e61-b04f-01a993b10388\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " Oct 30 21:53:22.965196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.965097 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\") pod \"4ce2f42c-0d56-4e61-b04f-01a993b10388\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " Oct 30 21:53:22.965292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.965245 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\") pod \"4ce2f42c-0d56-4e61-b04f-01a993b10388\" (UID: \"4ce2f42c-0d56-4e61-b04f-01a993b10388\") " Oct 30 21:53:22.965622 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.965592 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca" pod "4ce2f42c-0d56-4e61-b04f-01a993b10388" (UID: "4ce2f42c-0d56-4e61-b04f-01a993b10388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:22.965861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.965840 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:22.968146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.968120 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca" pod "4ce2f42c-0d56-4e61-b04f-01a993b10388" (UID: "4ce2f42c-0d56-4e61-b04f-01a993b10388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:22.971169 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:22.971137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz" pod "4ce2f42c-0d56-4e61-b04f-01a993b10388" (UID: "4ce2f42c-0d56-4e61-b04f-01a993b10388"). InnerVolumeSpecName "kube-api-access-qdxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:23.067275 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.067231 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce2f42c-0d56-4e61-b04f-01a993b10388-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:23.067390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.067283 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxhz\" (UniqueName: \"kubernetes.io/projected/4ce2f42c-0d56-4e61-b04f-01a993b10388-kube-api-access-qdxhz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:23.210629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.210594 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:23.212528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.212492 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:23.212669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.212558 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:23.759691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.759660 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" Oct 30 21:53:23.759896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.759698 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf" event={"ID":"4ce2f42c-0d56-4e61-b04f-01a993b10388","Type":"ContainerDied","Data":"9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b"} Oct 30 21:53:23.759896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.759753 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a68f44a0ace283e2198757de16e23ef7a5dc3fd33559e01c6e227ce4a3f818b" Oct 30 21:53:23.991401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:23.991371 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:24.078061 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.077978 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\") pod \"5661e199-85a5-4a15-9ad5-720233255d95\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " Oct 30 21:53:24.078236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.078086 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\" (UniqueName: \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\") pod \"5661e199-85a5-4a15-9ad5-720233255d95\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " Oct 30 21:53:24.078236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.078198 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\") pod \"5661e199-85a5-4a15-9ad5-720233255d95\" (UID: \"5661e199-85a5-4a15-9ad5-720233255d95\") " Oct 30 21:53:24.079136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.079105 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca" pod "5661e199-85a5-4a15-9ad5-720233255d95" (UID: "5661e199-85a5-4a15-9ad5-720233255d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:24.081920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.081893 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x" pod "5661e199-85a5-4a15-9ad5-720233255d95" (UID: "5661e199-85a5-4a15-9ad5-720233255d95"). InnerVolumeSpecName "kube-api-access-9zf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:24.082399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.082373 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca" pod "5661e199-85a5-4a15-9ad5-720233255d95" (UID: "5661e199-85a5-4a15-9ad5-720233255d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:24.191298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.188903 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zf6x\" (UniqueName: \"kubernetes.io/projected/5661e199-85a5-4a15-9ad5-720233255d95-kube-api-access-9zf6x\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:24.191298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.188937 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:24.191298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.188952 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5661e199-85a5-4a15-9ad5-720233255d95-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:24.246288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.242727 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:24.246288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.242808 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:24.768399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.768359 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" event={"ID":"5661e199-85a5-4a15-9ad5-720233255d95","Type":"ContainerDied","Data":"6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933"} Oct 30 21:53:24.768590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.768419 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5bd017f99ce27197e23d5e64fee15c46c38209dd54c217eb1d3df029300933" Oct 30 21:53:24.768590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:24.768372 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m" Oct 30 21:53:25.218292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:25.210321 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:25.218292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:25.210387 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:26.223014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:26.222983 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:26.234402 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:26.234360 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:27.222839 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:27.222800 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:27.223139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:27.223102 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:28.230292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:28.229373 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:28.230292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:28.229431 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:29.210237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.210202 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:53:29.210456 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.210254 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:53:29.616872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.616782 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb"] Oct 30 21:53:29.617661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617639 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5661e199-85a5-4a15-9ad5-720233255d95" containerName="revoke-keys" Oct 30 21:53:29.617753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617662 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5661e199-85a5-4a15-9ad5-720233255d95" containerName="revoke-keys" Oct 30 21:53:29.617753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617702 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4ce2f42c-0d56-4e61-b04f-01a993b10388" containerName="revoke-keys" Oct 30 21:53:29.617753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617711 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce2f42c-0d56-4e61-b04f-01a993b10388" containerName="revoke-keys" Oct 30 21:53:29.617916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617841 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce2f42c-0d56-4e61-b04f-01a993b10388" containerName="revoke-keys" Oct 30 21:53:29.617916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.617859 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5661e199-85a5-4a15-9ad5-720233255d95" containerName="revoke-keys" Oct 30 21:53:29.622094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.622072 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.630303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.626206 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:29.630303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.626892 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:29.630303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.626987 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:29.630303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.629291 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb"] Oct 30 21:53:29.660129 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.660097 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnvz\" (UniqueName: \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.660293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.660151 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.660382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.660356 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.725594 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.725566 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:29.761879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.761847 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnvz\" (UniqueName: \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.762049 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.761902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.762102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.762056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.762689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.762665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.764391 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.764373 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.773713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.773693 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnvz\" (UniqueName: \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\") pod \"maas-api-revoke-sub-66328687-g7blb\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:29.945707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:29.945666 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:30.257678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.257644 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb"] Oct 30 21:53:30.261321 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:30.260791 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591cdea1_d8b7_47ae_b262_1850c5966742.slice/crio-35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a WatchSource:0}: Error finding container 35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a: Status 404 returned error can't find the container with id 35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a Oct 30 21:53:30.433592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.432320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:30.433592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.433458 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:30.434159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.434141 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 21:53:30.843303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.843238 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw"] Oct 30 21:53:30.848024 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.848001 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.858510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.858483 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw"] Oct 30 21:53:30.876856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.876768 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.877018 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.876877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmlf\" (UniqueName: \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.877120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.877083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.889547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.889509 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" event={"ID":"591cdea1-d8b7-47ae-b262-1850c5966742","Type":"ContainerStarted","Data":"dc297f16ce28f294703398cba68800d0746d2fcd985ef7e3c82c72e4d508615a"} Oct 30 21:53:30.889695 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.889564 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" event={"ID":"591cdea1-d8b7-47ae-b262-1850c5966742","Type":"ContainerStarted","Data":"35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a"} Oct 30 21:53:30.910412 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.910357 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" podStartSLOduration=1.910338743 podStartE2EDuration="1.910338743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:30.908061416 +0000 UTC m=+834.858016361" watchObservedRunningTime="2026-10-30 21:53:30.910338743 +0000 UTC m=+834.860293690" Oct 30 21:53:30.983319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.983285 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmlf\" (UniqueName: \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.985660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.985632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.991831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.991802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:30.996938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:30.992830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:31.046996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.046943 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmlf\" (UniqueName: \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:31.048721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.047566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d8a522-w4jcw\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:31.170287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.170230 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:31.585569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.582347 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw"] Oct 30 21:53:31.896616 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.896574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" event={"ID":"6078ee51-072e-4815-b65c-60b05f4cf6bd","Type":"ContainerStarted","Data":"46198c3ad2f56f0f67b4d59a344804d481e20d009a40d5fa8b8faa1fc9e2f916"} Oct 30 21:53:31.898360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.898332 5036 generic.go:356] "Generic (PLEG): container finished" podID="591cdea1-d8b7-47ae-b262-1850c5966742" containerID="dc297f16ce28f294703398cba68800d0746d2fcd985ef7e3c82c72e4d508615a" exitCode=0 Oct 30 21:53:31.898453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:31.898383 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" event={"ID":"591cdea1-d8b7-47ae-b262-1850c5966742","Type":"ContainerDied","Data":"dc297f16ce28f294703398cba68800d0746d2fcd985ef7e3c82c72e4d508615a"} Oct 30 21:53:32.909130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:32.909096 5036 generic.go:356] "Generic (PLEG): container finished" podID="6078ee51-072e-4815-b65c-60b05f4cf6bd" containerID="acb4d691042a58fd97ef1c924d040d8e91d1a2084ceb1899acda7eeecd93fa63" exitCode=0 Oct 30 21:53:32.909614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:32.909180 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" event={"ID":"6078ee51-072e-4815-b65c-60b05f4cf6bd","Type":"ContainerDied","Data":"acb4d691042a58fd97ef1c924d040d8e91d1a2084ceb1899acda7eeecd93fa63"} Oct 30 21:53:33.071657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.071631 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:33.130798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.130764 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\" (UniqueName: \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\") pod \"591cdea1-d8b7-47ae-b262-1850c5966742\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " Oct 30 21:53:33.130981 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.130874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\") pod \"591cdea1-d8b7-47ae-b262-1850c5966742\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " Oct 30 21:53:33.131025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.131003 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\") pod \"591cdea1-d8b7-47ae-b262-1850c5966742\" (UID: \"591cdea1-d8b7-47ae-b262-1850c5966742\") " Oct 30 21:53:33.131237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.131212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca" pod "591cdea1-d8b7-47ae-b262-1850c5966742" (UID: "591cdea1-d8b7-47ae-b262-1850c5966742"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:33.131475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.131451 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.133351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.133322 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz" pod "591cdea1-d8b7-47ae-b262-1850c5966742" (UID: "591cdea1-d8b7-47ae-b262-1850c5966742"). InnerVolumeSpecName "kube-api-access-jwnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:33.135375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.135352 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca" pod "591cdea1-d8b7-47ae-b262-1850c5966742" (UID: "591cdea1-d8b7-47ae-b262-1850c5966742"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:33.233035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.232951 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591cdea1-d8b7-47ae-b262-1850c5966742-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.233035 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.232980 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnvz\" (UniqueName: \"kubernetes.io/projected/591cdea1-d8b7-47ae-b262-1850c5966742-kube-api-access-jwnvz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.918933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.918908 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" Oct 30 21:53:33.919350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.918947 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb" event={"ID":"591cdea1-d8b7-47ae-b262-1850c5966742","Type":"ContainerDied","Data":"35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a"} Oct 30 21:53:33.919350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:33.919006 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ec4a4f54c66beda70a56b63f2cd3b3d3bf8f75ad7da2860d68d2804deed30a" Oct 30 21:53:34.092508 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.092478 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:34.149008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.148974 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\" (UniqueName: \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\") pod \"6078ee51-072e-4815-b65c-60b05f4cf6bd\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " Oct 30 21:53:34.149246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.149084 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\") pod \"6078ee51-072e-4815-b65c-60b05f4cf6bd\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " Oct 30 21:53:34.149246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.149170 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\") pod \"6078ee51-072e-4815-b65c-60b05f4cf6bd\" (UID: \"6078ee51-072e-4815-b65c-60b05f4cf6bd\") " Oct 30 21:53:34.149802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.149777 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca" pod "6078ee51-072e-4815-b65c-60b05f4cf6bd" (UID: "6078ee51-072e-4815-b65c-60b05f4cf6bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:34.150007 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.149983 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:34.156482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.156449 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca" pod "6078ee51-072e-4815-b65c-60b05f4cf6bd" (UID: "6078ee51-072e-4815-b65c-60b05f4cf6bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:34.156701 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.156675 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf" pod "6078ee51-072e-4815-b65c-60b05f4cf6bd" (UID: "6078ee51-072e-4815-b65c-60b05f4cf6bd"). InnerVolumeSpecName "kube-api-access-wlmlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:34.251839 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.251805 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6078ee51-072e-4815-b65c-60b05f4cf6bd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:34.251839 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.251840 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmlf\" (UniqueName: \"kubernetes.io/projected/6078ee51-072e-4815-b65c-60b05f4cf6bd-kube-api-access-wlmlf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:34.951857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.951707 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" event={"ID":"6078ee51-072e-4815-b65c-60b05f4cf6bd","Type":"ContainerDied","Data":"46198c3ad2f56f0f67b4d59a344804d481e20d009a40d5fa8b8faa1fc9e2f916"} Oct 30 21:53:34.951857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.951780 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46198c3ad2f56f0f67b4d59a344804d481e20d009a40d5fa8b8faa1fc9e2f916" Oct 30 21:53:34.951857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:34.951716 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw" Oct 30 21:53:35.443326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.443225 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 21:53:35.444181 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444160 5036 cpu_manager.go:414] "containerMap: removing container" podUID="591cdea1-d8b7-47ae-b262-1850c5966742" containerName="revoke-keys" Oct 30 21:53:35.444308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444183 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591cdea1-d8b7-47ae-b262-1850c5966742" containerName="revoke-keys" Oct 30 21:53:35.444308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444222 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6078ee51-072e-4815-b65c-60b05f4cf6bd" containerName="revoke-keys" Oct 30 21:53:35.444308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444229 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6078ee51-072e-4815-b65c-60b05f4cf6bd" containerName="revoke-keys" Oct 30 21:53:35.444411 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444396 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="591cdea1-d8b7-47ae-b262-1850c5966742" containerName="revoke-keys" Oct 30 21:53:35.444455 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.444412 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6078ee51-072e-4815-b65c-60b05f4cf6bd" containerName="revoke-keys" Oct 30 21:53:35.448716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.448689 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.459709 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.459684 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"distinct-w2-2b2e72-kserve-self-signed-certs\"" Oct 30 21:53:35.460023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.459697 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 21:53:35.567790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.567748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.567982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.567825 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.567982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.567924 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.567982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.567963 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.568124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.568015 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.568185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.568125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzgl\" (UniqueName: \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669394 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669586 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669444 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669586 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669586 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669552 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzgl\" (UniqueName: \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669795 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669595 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669795 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669627 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.669946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669915 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.670028 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.669959 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.670077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.670042 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.672058 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.672029 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.672320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.672305 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.682436 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.682407 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzgl\" (UniqueName: \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\") pod \"distinct-w2-2b2e72-kserve-669fc48dcf-jknzt\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:35.774108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:35.774011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:36.069890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:36.069858 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 21:53:36.070710 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:36.070686 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9f550c0_8573_46f1_9507_3d16da78855c.slice/crio-c902655f3e9dd371166e940fce35107a89333883be48bc085925a032d1382944 WatchSource:0}: Error finding container c902655f3e9dd371166e940fce35107a89333883be48bc085925a032d1382944: Status 404 returned error can't find the container with id c902655f3e9dd371166e940fce35107a89333883be48bc085925a032d1382944 Oct 30 21:53:36.968687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:36.968650 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" event={"ID":"e9f550c0-8573-46f1-9507-3d16da78855c","Type":"ContainerStarted","Data":"25bfff488cc8ae97e6f99adcae1fadc08bd76934c978e2193a8cd8169954eb0b"} Oct 30 21:53:36.968842 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:36.968701 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" event={"ID":"e9f550c0-8573-46f1-9507-3d16da78855c","Type":"ContainerStarted","Data":"c902655f3e9dd371166e940fce35107a89333883be48bc085925a032d1382944"} Oct 30 21:53:36.994388 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:36.994337 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podStartSLOduration=1.994325514 podStartE2EDuration="1.994325514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:36.990574984 +0000 UTC m=+840.940529930" watchObservedRunningTime="2026-10-30 21:53:36.994325514 +0000 UTC m=+840.944280460" Oct 30 21:53:37.774133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:37.774101 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:41.721292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:41.717947 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:53:41.738293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:41.736573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:41.875292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:41.875033 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:53:42.016365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:41.998777 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.020290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.018952 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 21:53:42.117290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.102398 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.721397 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.721346 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:53:42.723475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.723450 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:53:42.746444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.746407 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.747013 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.746991 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:53:42.876139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.876093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.877659 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:42.877632 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:53:43.012921 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.012683 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:53:43.062730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.051336 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:53:43.083036 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.083011 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:53:43.231289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.230067 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:53:43.295301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.285731 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:53:43.295301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.295144 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:53:43.673374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.673337 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4"] Oct 30 21:53:43.678926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.678910 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.688993 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.688970 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4"] Oct 30 21:53:43.692425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.692400 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:43.692574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.692438 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:43.692574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.692540 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:43.692574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.692545 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:43.725753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.725725 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.726350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.725898 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlr7\" (UniqueName: \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.726350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.725939 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.827032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.826995 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlr7\" (UniqueName: \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.827174 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.827051 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.827174 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.827135 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.827721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.827693 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.829715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.829690 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:43.839729 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:43.839704 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlr7\" (UniqueName: \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\") pod \"maas-api-revoke-sub-51672719-7bdt4\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:44.024345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.024245 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:44.317901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.317873 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4"] Oct 30 21:53:44.319195 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:44.319171 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fec8940_34af_43ed_9bf9_b7c96c212e2c.slice/crio-0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1 WatchSource:0}: Error finding container 0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1: Status 404 returned error can't find the container with id 0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1 Oct 30 21:53:44.508214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.508185 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" event={"ID":"4fec8940-34af-43ed-9bf9-b7c96c212e2c","Type":"ContainerStarted","Data":"0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1"} Oct 30 21:53:44.723956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.723919 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg"] Oct 30 21:53:44.728141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.728117 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.767948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.767920 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg"] Oct 30 21:53:44.839236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.839201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4tm\" (UniqueName: \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.839404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.839245 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.839404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.839304 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.940613 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.940579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4tm\" (UniqueName: \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.940778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.940622 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.940778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.940652 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.941324 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.941301 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.943217 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.943191 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:44.953838 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:44.953812 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4tm\" (UniqueName: \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\") pod \"maas-api-revoke-sub-6d3ac509-bfcjg\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:45.042404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.042373 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:45.339634 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:45.339606 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a13ed6_6da9_4a93_a463_8d1ea7506084.slice/crio-3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c WatchSource:0}: Error finding container 3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c: Status 404 returned error can't find the container with id 3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c Oct 30 21:53:45.342085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.342058 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg"] Oct 30 21:53:45.519230 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.519153 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" event={"ID":"62a13ed6-6da9-4a93-a463-8d1ea7506084","Type":"ContainerStarted","Data":"3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c"} Oct 30 21:53:45.520924 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.520895 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fec8940-34af-43ed-9bf9-b7c96c212e2c" containerID="3013e0228b1198a736eee9f4172dc8c06d6870ccc21fbccf4cc3a357b153cd4e" exitCode=0 Oct 30 21:53:45.521041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.520939 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" event={"ID":"4fec8940-34af-43ed-9bf9-b7c96c212e2c","Type":"ContainerDied","Data":"3013e0228b1198a736eee9f4172dc8c06d6870ccc21fbccf4cc3a357b153cd4e"} Oct 30 21:53:45.775085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.774979 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:45.791699 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:45.791674 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:46.533631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.533597 5036 generic.go:356] "Generic (PLEG): container finished" podID="62a13ed6-6da9-4a93-a463-8d1ea7506084" containerID="3e8618f54b3bb858b0872734f4e2c96d1cba2e33832205c17f1328ddaca580fa" exitCode=0 Oct 30 21:53:46.533796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.533688 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" event={"ID":"62a13ed6-6da9-4a93-a463-8d1ea7506084","Type":"ContainerDied","Data":"3e8618f54b3bb858b0872734f4e2c96d1cba2e33832205c17f1328ddaca580fa"} Oct 30 21:53:46.555641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.555611 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:53:46.743158 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.743130 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:46.867858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.867827 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\" (UniqueName: \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\") pod \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " Oct 30 21:53:46.868385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.867874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\") pod \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " Oct 30 21:53:46.868385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.867952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\") pod \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\" (UID: \"4fec8940-34af-43ed-9bf9-b7c96c212e2c\") " Oct 30 21:53:46.868385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.868279 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca" pod "4fec8940-34af-43ed-9bf9-b7c96c212e2c" (UID: "4fec8940-34af-43ed-9bf9-b7c96c212e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:46.868599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.868581 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:46.870228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.870205 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca" pod "4fec8940-34af-43ed-9bf9-b7c96c212e2c" (UID: "4fec8940-34af-43ed-9bf9-b7c96c212e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:46.870338 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.870314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7" pod "4fec8940-34af-43ed-9bf9-b7c96c212e2c" (UID: "4fec8940-34af-43ed-9bf9-b7c96c212e2c"). InnerVolumeSpecName "kube-api-access-vvlr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:46.969890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.969808 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlr7\" (UniqueName: \"kubernetes.io/projected/4fec8940-34af-43ed-9bf9-b7c96c212e2c-kube-api-access-vvlr7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:46.969890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:46.969836 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fec8940-34af-43ed-9bf9-b7c96c212e2c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:47.545097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.545067 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" Oct 30 21:53:47.545097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.545090 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4" event={"ID":"4fec8940-34af-43ed-9bf9-b7c96c212e2c","Type":"ContainerDied","Data":"0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1"} Oct 30 21:53:47.545344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.545146 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5887e190db9a2910868b40e618633e9dba75ccd29faf2501d0cd7893a129b1" Oct 30 21:53:47.701594 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.701564 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:47.780131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.780099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\") pod \"62a13ed6-6da9-4a93-a463-8d1ea7506084\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " Oct 30 21:53:47.780350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.780140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\") pod \"62a13ed6-6da9-4a93-a463-8d1ea7506084\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " Oct 30 21:53:47.780350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.780247 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\" (UniqueName: \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\") pod \"62a13ed6-6da9-4a93-a463-8d1ea7506084\" (UID: \"62a13ed6-6da9-4a93-a463-8d1ea7506084\") " Oct 30 21:53:47.780491 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.780461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca" pod "62a13ed6-6da9-4a93-a463-8d1ea7506084" (UID: "62a13ed6-6da9-4a93-a463-8d1ea7506084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:47.780723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.780707 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:47.782482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.782439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca" pod "62a13ed6-6da9-4a93-a463-8d1ea7506084" (UID: "62a13ed6-6da9-4a93-a463-8d1ea7506084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:47.782573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.782523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm" pod "62a13ed6-6da9-4a93-a463-8d1ea7506084" (UID: "62a13ed6-6da9-4a93-a463-8d1ea7506084"). InnerVolumeSpecName "kube-api-access-ql4tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:47.882118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.882088 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4tm\" (UniqueName: \"kubernetes.io/projected/62a13ed6-6da9-4a93-a463-8d1ea7506084-kube-api-access-ql4tm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:47.882475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:47.882119 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a13ed6-6da9-4a93-a463-8d1ea7506084-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:48.103633 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:53:48.103594 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:53:48.203308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.203278 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5"] Oct 30 21:53:48.203906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.203887 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fec8940-34af-43ed-9bf9-b7c96c212e2c" containerName="revoke-keys" Oct 30 21:53:48.203997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.203908 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fec8940-34af-43ed-9bf9-b7c96c212e2c" containerName="revoke-keys" Oct 30 21:53:48.203997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.203938 5036 cpu_manager.go:414] "containerMap: removing container" podUID="62a13ed6-6da9-4a93-a463-8d1ea7506084" containerName="revoke-keys" Oct 30 21:53:48.203997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.203948 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a13ed6-6da9-4a93-a463-8d1ea7506084" containerName="revoke-keys" Oct 30 21:53:48.204143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.204080 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a13ed6-6da9-4a93-a463-8d1ea7506084" containerName="revoke-keys" Oct 30 21:53:48.204143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.204104 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fec8940-34af-43ed-9bf9-b7c96c212e2c" containerName="revoke-keys" Oct 30 21:53:48.207716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.207698 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.211715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.211681 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lx85x\"" Oct 30 21:53:48.212079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.212057 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:53:48.222678 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.222658 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5"] Oct 30 21:53:48.286616 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.286581 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.286800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.286715 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.286800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.286765 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tk6q\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-kube-api-access-4tk6q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.286886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.286870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.287150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.287127 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.287308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.287254 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.287469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.287446 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.311434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.311399 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:53:48.311593 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.311451 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:53:48.388931 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.388901 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.388931 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.388946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.388968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tk6q\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-kube-api-access-4tk6q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.389011 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.389062 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.389105 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.389142 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.389833 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.389801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.391469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.391433 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.391661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.391639 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.391852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.391830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.392151 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.392131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.402162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.402134 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.402314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.402247 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tk6q\" (UniqueName: \"kubernetes.io/projected/724f765c-ab49-4da9-9ce2-f94dd12bb2b3-kube-api-access-4tk6q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ggxs5\" (UID: \"724f765c-ab49-4da9-9ce2-f94dd12bb2b3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.524218 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.524118 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:48.557198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.557169 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" Oct 30 21:53:48.557579 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.557192 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg" event={"ID":"62a13ed6-6da9-4a93-a463-8d1ea7506084","Type":"ContainerDied","Data":"3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c"} Oct 30 21:53:48.560077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.557783 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3874455007a33b3013667d717d8b64f04a1c9803b94321a9752d936ca14dc96c" Oct 30 21:53:48.836744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:48.836710 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5"] Oct 30 21:53:48.837457 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:48.837414 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724f765c_ab49_4da9_9ce2_f94dd12bb2b3.slice/crio-8610222127bc28a59f1194a3794f9e04678d84ff07b271bc89ecc67353e6cefa WatchSource:0}: Error finding container 8610222127bc28a59f1194a3794f9e04678d84ff07b271bc89ecc67353e6cefa: Status 404 returned error can't find the container with id 8610222127bc28a59f1194a3794f9e04678d84ff07b271bc89ecc67353e6cefa Oct 30 21:53:49.567015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:49.566979 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" event={"ID":"724f765c-ab49-4da9-9ce2-f94dd12bb2b3","Type":"ContainerStarted","Data":"8610222127bc28a59f1194a3794f9e04678d84ff07b271bc89ecc67353e6cefa"} Oct 30 21:53:52.012569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.012536 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 21:53:52.435242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.435210 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 21:53:52.441298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.438133 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.584235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.584196 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.584560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.584534 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.584717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.584697 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6qz\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.584896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.584877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.585053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.585035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.585214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.585198 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.589528 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.589504 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.589721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.589702 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.589893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.589874 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.691413 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.691311 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.692108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.692083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.692656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.692636 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.693130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.693109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6qz\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.694155 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.694134 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.695351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.695329 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.695833 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.695812 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.696097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.696079 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.696237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.696221 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.696445 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.695734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.696568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.693055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.697388 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.694576 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.698298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.692006 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.698439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.698199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.726124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.717000 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.730848 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.727887 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.730848 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.729120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.746786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.731707 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6qz\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:52.918723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:52.918430 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:53.246703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.246673 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 21:53:53.248538 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:53:53.248507 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dcda32_bf9b_4260_875e_0eceacf7be36.slice/crio-45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f WatchSource:0}: Error finding container 45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f: Status 404 returned error can't find the container with id 45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f Oct 30 21:53:53.251213 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.251183 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:53.251321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.251245 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:53.251391 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.251363 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:53.683578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.683539 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" event={"ID":"47dcda32-bf9b-4260-875e-0eceacf7be36","Type":"ContainerStarted","Data":"21fdc22a7972093f5f31d43218a897f67d1112b17fdcfc46d5c710a93ba5186f"} Oct 30 21:53:53.683759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.683612 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" event={"ID":"47dcda32-bf9b-4260-875e-0eceacf7be36","Type":"ContainerStarted","Data":"45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f"} Oct 30 21:53:53.720802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.720747 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podStartSLOduration=2.720728461 podStartE2EDuration="2.720728461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:53.717327547 +0000 UTC m=+857.667282494" watchObservedRunningTime="2026-10-30 21:53:53.720728461 +0000 UTC m=+857.670683413" Oct 30 21:53:53.919319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:53.919242 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:53:54.908731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:54.908689 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:54.909051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:54.908753 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:54.929290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:54.925557 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:54.929290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:54.925615 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:53:55.704514 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:55.704469 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" event={"ID":"724f765c-ab49-4da9-9ce2-f94dd12bb2b3","Type":"ContainerStarted","Data":"7adfd359ee175ce520a7f55ce502ca3c22c78cc898f470e3b4b3bfb9a987d9b7"} Oct 30 21:53:55.704735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:55.704645 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:55.741334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:55.734226 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" podStartSLOduration=1.666756923 podStartE2EDuration="7.734204556s" totalImagesPullingTime="6.067447633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:55.731780243 +0000 UTC m=+859.681735191" watchObservedRunningTime="2026-10-30 21:53:55.734204556 +0000 UTC m=+859.684159503" Oct 30 21:53:55.923291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:55.922674 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:55.923291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:55.922736 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:53:56.742922 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.742887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ggxs5" Oct 30 21:53:56.863717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.863683 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k"] Oct 30 21:53:56.868254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.868233 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.880845 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.880820 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k"] Oct 30 21:53:56.894023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.891247 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67"] Oct 30 21:53:56.894023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.893710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.894023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.893746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.894023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.893979 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xznh\" (UniqueName: \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.895505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.895486 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:56.895783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.895536 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:56.895783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.895608 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:56.895783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.895601 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:53:56.896479 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.896463 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:56.940295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.939593 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:56.940295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.939644 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:53:56.977359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.977325 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67"] Oct 30 21:53:56.995189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995113 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xznh\" (UniqueName: \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.995189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995158 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:56.995435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995199 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.995435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995214 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:56.995435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995420 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bdh\" (UniqueName: \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:56.995597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.995462 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:56.996178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.996162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:57.000252 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:56.997771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:57.032291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.025968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xznh\" (UniqueName: \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\") pod \"maas-api-revoke-sub-a715cd9e-9lc9k\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:57.096883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.096846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.097067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.097047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bdh\" (UniqueName: \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.097155 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.097086 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.097910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.097807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.099831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.099808 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.141149 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.141100 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bdh\" (UniqueName: \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\") pod \"maas-api-revoke-sub-a958dcf5-rbp67\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.196145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.196122 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:53:57.280044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.279971 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:53:57.551645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.551532 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k"] Oct 30 21:53:57.705457 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.705430 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67"] Oct 30 21:53:57.752246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.752216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" event={"ID":"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd","Type":"ContainerStarted","Data":"b34e85f2d5d5ea2299b727721f84f25af3c4e8821a40f836cafd040395c2cebe"} Oct 30 21:53:57.930313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.927396 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:57.930313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:57.927475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:53:58.763000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.762957 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" event={"ID":"17c79123-31fd-4e2f-8e45-aed9939d834d","Type":"ContainerStarted","Data":"5765e8d55dc4360d5b4c81b2bd996aaccfab0e5d13d7cc38277799fcb38829ba"} Oct 30 21:53:58.763527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.763338 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" event={"ID":"17c79123-31fd-4e2f-8e45-aed9939d834d","Type":"ContainerStarted","Data":"5284f62de30e935eab23c40338ad1f88954a739fe642f723151d4899f6ae4fb8"} Oct 30 21:53:58.765870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.765838 5036 generic.go:356] "Generic (PLEG): container finished" podID="f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" containerID="7ee2134d705528b20c2e31c89cefb900a7e61b3b1c78014aeee04fdd36305432" exitCode=0 Oct 30 21:53:58.766009 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.765899 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" event={"ID":"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd","Type":"ContainerDied","Data":"7ee2134d705528b20c2e31c89cefb900a7e61b3b1c78014aeee04fdd36305432"} Oct 30 21:53:58.847289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.845814 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" podStartSLOduration=2.845795268 podStartE2EDuration="2.845795268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:58.797862059 +0000 UTC m=+862.747817005" watchObservedRunningTime="2026-10-30 21:53:58.845795268 +0000 UTC m=+862.795750214" Oct 30 21:53:58.926102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.925655 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:58.926102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:58.925721 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:53:59.775328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:59.775294 5036 generic.go:356] "Generic (PLEG): container finished" podID="17c79123-31fd-4e2f-8e45-aed9939d834d" containerID="5765e8d55dc4360d5b4c81b2bd996aaccfab0e5d13d7cc38277799fcb38829ba" exitCode=0 Oct 30 21:53:59.775723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:59.775368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" event={"ID":"17c79123-31fd-4e2f-8e45-aed9939d834d","Type":"ContainerDied","Data":"5765e8d55dc4360d5b4c81b2bd996aaccfab0e5d13d7cc38277799fcb38829ba"} Oct 30 21:53:59.919899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:59.919861 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:53:59.920101 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:53:59.919951 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:54:00.004723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.004695 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:54:00.183770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.183741 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\") pod \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " Oct 30 21:54:00.183963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.183803 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\") pod \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " Oct 30 21:54:00.183963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.183838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\" (UniqueName: \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\") pod \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\" (UID: \"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd\") " Oct 30 21:54:00.184159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.184135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca" pod "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" (UID: "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:00.184306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.184258 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.186340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.186316 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh" pod "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" (UID: "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd"). InnerVolumeSpecName "kube-api-access-6xznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:00.186456 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.186381 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca" pod "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" (UID: "f14c2347-020d-4ae6-b2fd-8e00efcdb5dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:00.286880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.286842 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.286880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.286882 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xznh\" (UniqueName: \"kubernetes.io/projected/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd-kube-api-access-6xznh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.789304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.785942 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" Oct 30 21:54:00.789304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.785980 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k" event={"ID":"f14c2347-020d-4ae6-b2fd-8e00efcdb5dd","Type":"ContainerDied","Data":"b34e85f2d5d5ea2299b727721f84f25af3c4e8821a40f836cafd040395c2cebe"} Oct 30 21:54:00.789304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:00.786025 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34e85f2d5d5ea2299b727721f84f25af3c4e8821a40f836cafd040395c2cebe" Oct 30 21:54:01.004748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.004707 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:54:01.098788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.098696 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\") pod \"17c79123-31fd-4e2f-8e45-aed9939d834d\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " Oct 30 21:54:01.098961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.098798 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\") pod \"17c79123-31fd-4e2f-8e45-aed9939d834d\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " Oct 30 21:54:01.098961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.098832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\" (UniqueName: \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\") pod \"17c79123-31fd-4e2f-8e45-aed9939d834d\" (UID: \"17c79123-31fd-4e2f-8e45-aed9939d834d\") " Oct 30 21:54:01.099708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.099678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca" pod "17c79123-31fd-4e2f-8e45-aed9939d834d" (UID: "17c79123-31fd-4e2f-8e45-aed9939d834d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:01.104566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.104538 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca" pod "17c79123-31fd-4e2f-8e45-aed9939d834d" (UID: "17c79123-31fd-4e2f-8e45-aed9939d834d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:01.106796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.106772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh" pod "17c79123-31fd-4e2f-8e45-aed9939d834d" (UID: "17c79123-31fd-4e2f-8e45-aed9939d834d"). InnerVolumeSpecName "kube-api-access-v6bdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:01.200794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.200761 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:01.200794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.200790 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c79123-31fd-4e2f-8e45-aed9939d834d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:01.200794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.200800 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6bdh\" (UniqueName: \"kubernetes.io/projected/17c79123-31fd-4e2f-8e45-aed9939d834d-kube-api-access-v6bdh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:01.351769 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.351686 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:54:01.352318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.352290 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:54:01.353206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.353186 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 21:54:01.796369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.796338 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" Oct 30 21:54:01.796369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.796345 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67" event={"ID":"17c79123-31fd-4e2f-8e45-aed9939d834d","Type":"ContainerDied","Data":"5284f62de30e935eab23c40338ad1f88954a739fe642f723151d4899f6ae4fb8"} Oct 30 21:54:01.796890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:01.796403 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5284f62de30e935eab23c40338ad1f88954a739fe642f723151d4899f6ae4fb8" Oct 30 21:54:07.601248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.601210 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606178 5036 cpu_manager.go:414] "containerMap: removing container" podUID="17c79123-31fd-4e2f-8e45-aed9939d834d" containerName="revoke-keys" Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606205 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c79123-31fd-4e2f-8e45-aed9939d834d" containerName="revoke-keys" Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606224 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" containerName="revoke-keys" Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606231 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" containerName="revoke-keys" Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606480 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" containerName="revoke-keys" Oct 30 21:54:07.607286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.606496 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c79123-31fd-4e2f-8e45-aed9939d834d" containerName="revoke-keys" Oct 30 21:54:07.642294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.612012 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742505 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742584 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742763 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742789 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhr7l\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742830 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742868 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.745290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.742887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.860347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.833747 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 21:54:07.911292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.862860 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.960514 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.960475 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhr7l\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.965352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.960787 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.995576 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.995544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.995809 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.995787 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.996039 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.996020 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.996161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.996145 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:07.996313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.996295 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.031052 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.913334 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.030924 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:07.993330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.032192 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.035289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.032905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.121356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.083781 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.121356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.084149 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.121356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.084839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhr7l\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.121356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.106304 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.121356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.120357 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.158294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.133958 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.256294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.249664 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg"] Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.281740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.286549 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.286714 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.286807 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.287831 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:08.299841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.295366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.318190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.318152 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg"] Oct 30 21:54:08.319753 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.318492 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.348124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.338585 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.348124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.338696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdk9n\" (UniqueName: \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.348124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.338829 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.439292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.438502 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:08.477898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.442155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdk9n\" (UniqueName: \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.477898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.442292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.477898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.442395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.477898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.443123 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.477898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.466740 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.545293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.535360 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.545293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.538306 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdk9n\" (UniqueName: \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\") pod \"maas-api-revoke-sub-c1ed0a9e-zbjfg\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:08.654289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.653383 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:54:08.654289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.653471 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.681552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.680347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.691879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:08.691744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:09.250542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.221363 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.254670 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.255675 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.275634 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.276679 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.277627 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.302055 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.349734 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:09.355371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.350353 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.356922 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.356107 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:09.622763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.618842 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.622763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.620126 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:09.726359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.692745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.726359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.693152 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:09.810346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.800867 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:09.810346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.801472 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:09.836300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.831459 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:09.938365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.867758 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:09.938365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.867849 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:09.938365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:09.867894 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:10.062141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.061982 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 21:54:10.133627 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.133584 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:10.204670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.204644 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 21:54:10.273532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.273497 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 21:54:10.273698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.273519 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.273911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.273694 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:54:10.274007 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.273994 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:10.308455 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.308423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:10.308587 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.308517 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:10.321962 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.321393 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg"] Oct 30 21:54:10.452525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452488 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bjc\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452606 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452865 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452775 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452865 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452814 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452940 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.452967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.452962 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.453067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.453033 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.553920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.553883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.553920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.553921 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.553962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.553994 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554096 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554119 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554189 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554301 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bjc\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554580 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554487 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554787 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554913 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554932 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.554968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.554963 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.557509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.557466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.563375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.560308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.567979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.567958 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bjc\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.568336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.568318 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.636925 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.636894 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:10.895070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.895034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" event={"ID":"5a609d12-a259-43b0-8247-362e5cec0ee8","Type":"ContainerStarted","Data":"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5"} Oct 30 21:54:10.895566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.895100 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" event={"ID":"5a609d12-a259-43b0-8247-362e5cec0ee8","Type":"ContainerStarted","Data":"b30aa1b4bcffd55af2234014a63ad81ee5f77362dc1e62f55f56ca0d103be077"} Oct 30 21:54:10.896895 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.896869 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" event={"ID":"5b04ad9d-bb33-4039-a085-ca8ec7d0c320","Type":"ContainerStarted","Data":"e2efc80bbb7ffb3a186fa26038b7bb666289b868107fd0ff00d8270394ceb3ca"} Oct 30 21:54:10.897024 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.896923 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" event={"ID":"5b04ad9d-bb33-4039-a085-ca8ec7d0c320","Type":"ContainerStarted","Data":"c54cc7ba0e713dd1dc7aa4b5b155778960dd251115ba9423a926fb0274ce1232"} Oct 30 21:54:10.948304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.945149 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podStartSLOduration=3.945126965 podStartE2EDuration="3.945126965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.941567796 +0000 UTC m=+874.891522745" watchObservedRunningTime="2026-10-30 21:54:10.945126965 +0000 UTC m=+874.895081913" Oct 30 21:54:10.953899 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:54:10.953868 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62a658e_fda8_4360_b570_b2299f4c6b17.slice/crio-514eb95e25aa4b10bc9b3a6ecff8c2ae215b9e606ca8c2eb50e0d03a168c4d7a WatchSource:0}: Error finding container 514eb95e25aa4b10bc9b3a6ecff8c2ae215b9e606ca8c2eb50e0d03a168c4d7a: Status 404 returned error can't find the container with id 514eb95e25aa4b10bc9b3a6ecff8c2ae215b9e606ca8c2eb50e0d03a168c4d7a Oct 30 21:54:10.957187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.956238 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 21:54:10.961408 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.961373 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:10.961515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.961437 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:10.961515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.961466 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:10.966128 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:10.966087 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" podStartSLOduration=3.966074468 podStartE2EDuration="3.966074468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.963008138 +0000 UTC m=+874.912963083" watchObservedRunningTime="2026-10-30 21:54:10.966074468 +0000 UTC m=+874.916029413" Oct 30 21:54:11.439666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.439633 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:11.443800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.443403 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:11.443800 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.443461 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:11.906798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.906763 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" event={"ID":"c62a658e-fda8-4360-b570-b2299f4c6b17","Type":"ContainerStarted","Data":"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700"} Oct 30 21:54:11.908835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.908367 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" event={"ID":"c62a658e-fda8-4360-b570-b2299f4c6b17","Type":"ContainerStarted","Data":"514eb95e25aa4b10bc9b3a6ecff8c2ae215b9e606ca8c2eb50e0d03a168c4d7a"} Oct 30 21:54:11.909626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.909604 5036 generic.go:356] "Generic (PLEG): container finished" podID="5b04ad9d-bb33-4039-a085-ca8ec7d0c320" containerID="e2efc80bbb7ffb3a186fa26038b7bb666289b868107fd0ff00d8270394ceb3ca" exitCode=0 Oct 30 21:54:11.914290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.910258 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" event={"ID":"5b04ad9d-bb33-4039-a085-ca8ec7d0c320","Type":"ContainerDied","Data":"e2efc80bbb7ffb3a186fa26038b7bb666289b868107fd0ff00d8270394ceb3ca"} Oct 30 21:54:11.951292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:11.945930 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podStartSLOduration=2.945911169 podStartE2EDuration="2.945911169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.933307015 +0000 UTC m=+875.883261960" watchObservedRunningTime="2026-10-30 21:54:11.945911169 +0000 UTC m=+875.895866120" Oct 30 21:54:12.451291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:12.451032 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:12.451291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:12.451099 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:12.646293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:12.645348 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:12.646293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:12.645511 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:12.646293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:12.645558 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:13.307638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.307614 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:13.406914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.406885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\" (UniqueName: \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\") pod \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " Oct 30 21:54:13.407080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.406951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\") pod \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " Oct 30 21:54:13.407164 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.407137 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\") pod \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\" (UID: \"5b04ad9d-bb33-4039-a085-ca8ec7d0c320\") " Oct 30 21:54:13.407530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.407503 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca" pod "5b04ad9d-bb33-4039-a085-ca8ec7d0c320" (UID: "5b04ad9d-bb33-4039-a085-ca8ec7d0c320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:13.407766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.407747 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.409634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.409612 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n" pod "5b04ad9d-bb33-4039-a085-ca8ec7d0c320" (UID: "5b04ad9d-bb33-4039-a085-ca8ec7d0c320"). InnerVolumeSpecName "kube-api-access-tdk9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:13.417313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.410327 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca" pod "5b04ad9d-bb33-4039-a085-ca8ec7d0c320" (UID: "5b04ad9d-bb33-4039-a085-ca8ec7d0c320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:13.468621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.468430 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:13.468621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.468503 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:13.509389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.509361 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdk9n\" (UniqueName: \"kubernetes.io/projected/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-kube-api-access-tdk9n\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.509389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.509385 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04ad9d-bb33-4039-a085-ca8ec7d0c320-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.645300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.639620 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:13.645300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.639679 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:13.994139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.994061 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" Oct 30 21:54:13.995385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.995352 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg" event={"ID":"5b04ad9d-bb33-4039-a085-ca8ec7d0c320","Type":"ContainerDied","Data":"c54cc7ba0e713dd1dc7aa4b5b155778960dd251115ba9423a926fb0274ce1232"} Oct 30 21:54:13.995499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:13.995413 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54cc7ba0e713dd1dc7aa4b5b155778960dd251115ba9423a926fb0274ce1232" Oct 30 21:54:14.463298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:14.460652 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:14.463298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:14.460714 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:14.644291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:14.643994 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:14.644291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:14.644057 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:15.452290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:15.451876 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:15.452290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:15.451933 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:15.649290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:15.642681 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:15.649290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:15.642747 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:16.450291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.450077 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:16.450291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.450149 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:16.643294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.642603 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:16.643294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.642657 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:16.679657 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.679623 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5"] Oct 30 21:54:16.680335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.680315 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5b04ad9d-bb33-4039-a085-ca8ec7d0c320" containerName="revoke-keys" Oct 30 21:54:16.680335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.680332 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b04ad9d-bb33-4039-a085-ca8ec7d0c320" containerName="revoke-keys" Oct 30 21:54:16.680461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.680435 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b04ad9d-bb33-4039-a085-ca8ec7d0c320" containerName="revoke-keys" Oct 30 21:54:16.684133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.684114 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.698001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.697901 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5"] Oct 30 21:54:16.703691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.702990 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:16.703691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.703063 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:16.703691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.703127 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:16.703691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.703448 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:16.802834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.802797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm6h\" (UniqueName: \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.803007 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.802852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.803056 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.803042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.904365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.904318 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.904532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.904464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm6h\" (UniqueName: \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.904532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.904496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.905192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.905174 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.907088 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.907053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:16.956378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:16.954838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm6h\" (UniqueName: \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\") pod \"maas-api-revoke-sub-18ee3639-bbff5\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:17.006430 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.006400 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:17.407714 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:54:17.407685 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3139f67_5130_4a7a_b5d5_e2dd2ea36e5e.slice/crio-059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b WatchSource:0}: Error finding container 059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b: Status 404 returned error can't find the container with id 059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b Oct 30 21:54:17.410471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.410443 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5"] Oct 30 21:54:17.439208 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.439170 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:54:17.439365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.439241 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:54:17.637550 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.637516 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:54:17.637748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:17.637573 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 21:54:18.103067 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:54:18.102966 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:54:18.105624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.105584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" event={"ID":"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e","Type":"ContainerStarted","Data":"2d5bcfcc5515947d0d29cc94c59c5dae038cafddbefbd7c04b5874269d1c983b"} Oct 30 21:54:18.105744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.105644 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" event={"ID":"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e","Type":"ContainerStarted","Data":"059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b"} Oct 30 21:54:18.157605 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.157546 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" podStartSLOduration=2.157530643 podStartE2EDuration="2.157530643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:18.151528202 +0000 UTC m=+882.101483140" watchObservedRunningTime="2026-10-30 21:54:18.157530643 +0000 UTC m=+882.107485589" Oct 30 21:54:18.466944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.466912 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:18.467396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.467377 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:18.468243 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.468225 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 21:54:18.660998 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.660304 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:18.661167 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.661075 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:18.664730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:18.664709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 21:54:19.115630 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:19.115591 5036 generic.go:356] "Generic (PLEG): container finished" podID="c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" containerID="2d5bcfcc5515947d0d29cc94c59c5dae038cafddbefbd7c04b5874269d1c983b" exitCode=0 Oct 30 21:54:19.116132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:19.115672 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" event={"ID":"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e","Type":"ContainerDied","Data":"2d5bcfcc5515947d0d29cc94c59c5dae038cafddbefbd7c04b5874269d1c983b"} Oct 30 21:54:20.294668 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.294641 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:20.390948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.390086 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\" (UniqueName: \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\") pod \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " Oct 30 21:54:20.390948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.390179 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\") pod \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " Oct 30 21:54:20.390948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.390337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\") pod \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\" (UID: \"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e\") " Oct 30 21:54:20.390948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.390568 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca" pod "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" (UID: "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:20.390948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.390913 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.393310 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.393240 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca" pod "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" (UID: "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:20.393446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.393309 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h" pod "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" (UID: "c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e"). InnerVolumeSpecName "kube-api-access-gnm6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:20.492032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.491998 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.492032 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:20.492028 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnm6h\" (UniqueName: \"kubernetes.io/projected/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e-kube-api-access-gnm6h\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:21.135847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:21.135823 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" Oct 30 21:54:21.136057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:21.135861 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5" event={"ID":"c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e","Type":"ContainerDied","Data":"059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b"} Oct 30 21:54:21.136057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:21.135924 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059060bc771cb138ffe2fc476ddbedc316c693440d2cce23c20d8b339d00db6b" Oct 30 21:54:22.805151 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:22.805098 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:22.842492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:22.842425 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:23.031542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.031507 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 21:54:23.080598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.032740 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" containerName="revoke-keys" Oct 30 21:54:23.091387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.091351 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" containerName="revoke-keys" Oct 30 21:54:23.102489 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.102457 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" containerName="revoke-keys" Oct 30 21:54:23.169656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.169624 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.249359 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.266540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.266610 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.266639 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbjn\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.266694 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.267330 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.267920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.267951 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.268001 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.268067 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.286299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.280811 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:23.369574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.369480 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.370382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.370355 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.370575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.370556 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.370736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.370719 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.370876 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.370857 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.371008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.370992 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.371123 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.371108 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbjn\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.371251 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.371236 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.371568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.371546 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.399342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.372333 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:23.414658 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.372652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.424677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.424642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.450871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.438473 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.452105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.451509 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.452105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.452052 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.478752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.478721 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 21:54:23.496291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.485748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbjn\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.600294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.597763 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.645294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.635457 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.646090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.646024 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.745107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.744842 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:23.858614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.856372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:23.858614 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:23.857179 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:24.044299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.013904 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:24.279358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.246679 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:24.325326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.325231 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:24.713645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.713150 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 21:54:24.737620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.737055 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:24.737620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.737161 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:24.737620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:24.737297 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:25.598574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.598533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" event={"ID":"4e773594-23d9-405b-80ab-e55075d3a52e","Type":"ContainerStarted","Data":"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be"} Oct 30 21:54:25.599014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.598605 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" event={"ID":"4e773594-23d9-405b-80ab-e55075d3a52e","Type":"ContainerStarted","Data":"6012942d3540dc26b2b2e4e2ed30c812b08afd3c2347f8a3465992944bd8953f"} Oct 30 21:54:25.654291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.649687 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podStartSLOduration=3.649667425 podStartE2EDuration="3.649667425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:25.649593099 +0000 UTC m=+889.599548074" watchObservedRunningTime="2026-10-30 21:54:25.649667425 +0000 UTC m=+889.599622371" Oct 30 21:54:25.745619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.745586 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:25.747301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.747234 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:25.747455 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:25.747315 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:26.746710 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.746671 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:26.747228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.746760 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:26.955514 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.955479 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp"] Oct 30 21:54:26.959997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.959977 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:26.966638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.966617 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:26.969495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.969470 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:26.969618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.969571 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:26.969618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.969586 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:26.975154 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.975131 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp"] Oct 30 21:54:26.990111 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.990080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:26.990396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.990371 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg75m\" (UniqueName: \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:26.990500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:26.990470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.091314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.091200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg75m\" (UniqueName: \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.091314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.091244 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.091523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.091338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.091939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.091919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.093847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.093826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.144093 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.144058 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg75m\" (UniqueName: \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\") pod \"maas-api-revoke-sub-95e45bae-2pcqp\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.276584 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.276554 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:27.644910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.644882 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp"] Oct 30 21:54:27.647425 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:54:27.647390 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3efb4f7_a766_4a73_af33_2fc5903ddc55.slice/crio-3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322 WatchSource:0}: Error finding container 3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322: Status 404 returned error can't find the container with id 3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322 Oct 30 21:54:27.745748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.745718 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:27.745927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.745767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:27.807095 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.807065 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5"] Oct 30 21:54:27.821527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.821501 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:27.859280 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.859229 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5"] Oct 30 21:54:27.904550 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.904475 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:27.904971 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.904949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:27.905154 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.905139 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmk2z\" (UniqueName: \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:27.960432 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.960401 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp"] Oct 30 21:54:27.965240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.965220 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:27.998911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:27.998884 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp"] Oct 30 21:54:28.007039 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.006956 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97hq\" (UniqueName: \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.007039 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.006992 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.007253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.007144 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.007253 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.007187 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk2z\" (UniqueName: \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.007395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.007341 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.007450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.007395 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.008372 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.008335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.010072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.010051 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.060061 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.060028 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk2z\" (UniqueName: \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\") pod \"maas-api-revoke-sub-140b1cb9-6lcj5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.108672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.108639 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.108856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.108736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97hq\" (UniqueName: \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.108856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.108784 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.109497 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.109477 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.133681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.130258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.142252 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.142170 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97hq\" (UniqueName: \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\") pod \"maas-api-revoke-sub-d50257ce-ht6jp\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.172920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.172845 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:28.283394 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.280321 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:28.551937 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.551910 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5"] Oct 30 21:54:28.654499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.654462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" event={"ID":"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5","Type":"ContainerStarted","Data":"36f84665284df75ab0369503133758e00ccc45934cfa9b86eb38086041d80024"} Oct 30 21:54:28.656146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.656112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" event={"ID":"f3efb4f7-a766-4a73-af33-2fc5903ddc55","Type":"ContainerStarted","Data":"ca1dc89ed3321592743459e03dee08cce58b2024c188daddfac9704691da291d"} Oct 30 21:54:28.656283 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.656160 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" event={"ID":"f3efb4f7-a766-4a73-af33-2fc5903ddc55","Type":"ContainerStarted","Data":"3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322"} Oct 30 21:54:28.678799 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.678777 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp"] Oct 30 21:54:28.700168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.700114 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" podStartSLOduration=2.700098907 podStartE2EDuration="2.700098907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:28.699256268 +0000 UTC m=+892.649211214" watchObservedRunningTime="2026-10-30 21:54:28.700098907 +0000 UTC m=+892.650053961" Oct 30 21:54:28.745854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.745832 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:28.745910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:28.745874 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:29.690317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.689776 5036 generic.go:356] "Generic (PLEG): container finished" podID="f3efb4f7-a766-4a73-af33-2fc5903ddc55" containerID="ca1dc89ed3321592743459e03dee08cce58b2024c188daddfac9704691da291d" exitCode=0 Oct 30 21:54:29.690317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.689940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" event={"ID":"f3efb4f7-a766-4a73-af33-2fc5903ddc55","Type":"ContainerDied","Data":"ca1dc89ed3321592743459e03dee08cce58b2024c188daddfac9704691da291d"} Oct 30 21:54:29.694889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.694001 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" event={"ID":"ef49a238-073d-4ce2-9f7c-3b1391196bde","Type":"ContainerStarted","Data":"0d6ed918ca0bc60b633efd363e1d2e180e21e72b58e52073fd3c0a48cd1fa186"} Oct 30 21:54:29.694889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.694064 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" event={"ID":"ef49a238-073d-4ce2-9f7c-3b1391196bde","Type":"ContainerStarted","Data":"a42cc7da4b990a452847ee2683efd9cf3b0b9d3f379c23734f42334babddf16a"} Oct 30 21:54:29.699300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.699006 5036 generic.go:356] "Generic (PLEG): container finished" podID="1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" containerID="1565b7287ab0b534ec6fa7238f951f751acffd6ea69539817181418ddf60f873" exitCode=0 Oct 30 21:54:29.699300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.699068 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" event={"ID":"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5","Type":"ContainerDied","Data":"1565b7287ab0b534ec6fa7238f951f751acffd6ea69539817181418ddf60f873"} Oct 30 21:54:29.758600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.758566 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:29.758869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:29.758841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:30.708836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:30.708797 5036 generic.go:356] "Generic (PLEG): container finished" podID="ef49a238-073d-4ce2-9f7c-3b1391196bde" containerID="0d6ed918ca0bc60b633efd363e1d2e180e21e72b58e52073fd3c0a48cd1fa186" exitCode=0 Oct 30 21:54:30.709303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:30.708885 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" event={"ID":"ef49a238-073d-4ce2-9f7c-3b1391196bde","Type":"ContainerDied","Data":"0d6ed918ca0bc60b633efd363e1d2e180e21e72b58e52073fd3c0a48cd1fa186"} Oct 30 21:54:30.745829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:30.745793 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:30.746006 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:30.745857 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:31.048522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.048492 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:31.054204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.054166 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:31.163384 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163352 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\") pod \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " Oct 30 21:54:31.163588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\") pod \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " Oct 30 21:54:31.163588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163453 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\" (UniqueName: \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\") pod \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " Oct 30 21:54:31.163588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163524 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\") pod \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " Oct 30 21:54:31.163751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163643 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\" (UniqueName: \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\") pod \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\" (UID: \"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5\") " Oct 30 21:54:31.163751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\") pod \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\" (UID: \"f3efb4f7-a766-4a73-af33-2fc5903ddc55\") " Oct 30 21:54:31.163858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.163749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca" pod "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" (UID: "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:31.164157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.164134 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca" pod "f3efb4f7-a766-4a73-af33-2fc5903ddc55" (UID: "f3efb4f7-a766-4a73-af33-2fc5903ddc55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:31.164256 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.164193 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.166527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.166499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca" pod "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" (UID: "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:31.166642 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.166622 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m" pod "f3efb4f7-a766-4a73-af33-2fc5903ddc55" (UID: "f3efb4f7-a766-4a73-af33-2fc5903ddc55"). InnerVolumeSpecName "kube-api-access-wg75m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:31.166794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.166772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca" pod "f3efb4f7-a766-4a73-af33-2fc5903ddc55" (UID: "f3efb4f7-a766-4a73-af33-2fc5903ddc55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:31.166923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.166810 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z" pod "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" (UID: "1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5"). InnerVolumeSpecName "kube-api-access-hmk2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:31.265599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.265490 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmk2z\" (UniqueName: \"kubernetes.io/projected/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-kube-api-access-hmk2z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.265599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.265528 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.265599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.265555 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.265599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.265572 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg75m\" (UniqueName: \"kubernetes.io/projected/f3efb4f7-a766-4a73-af33-2fc5903ddc55-kube-api-access-wg75m\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.265599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.265586 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3efb4f7-a766-4a73-af33-2fc5903ddc55-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.720223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.720193 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" Oct 30 21:54:31.720223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.720199 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5" event={"ID":"1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5","Type":"ContainerDied","Data":"36f84665284df75ab0369503133758e00ccc45934cfa9b86eb38086041d80024"} Oct 30 21:54:31.720841 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.720260 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f84665284df75ab0369503133758e00ccc45934cfa9b86eb38086041d80024" Oct 30 21:54:31.721919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.721896 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" Oct 30 21:54:31.722046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.721937 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp" event={"ID":"f3efb4f7-a766-4a73-af33-2fc5903ddc55","Type":"ContainerDied","Data":"3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322"} Oct 30 21:54:31.722046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.721982 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb42b517352f8abd73dd65dbcbfd0489471e347bfc6c8cee3cff8fe6c734322" Oct 30 21:54:31.745720 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.745695 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:54:31.745863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.745741 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:54:31.881414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.881389 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:31.974535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.974426 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\" (UniqueName: \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\") pod \"ef49a238-073d-4ce2-9f7c-3b1391196bde\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " Oct 30 21:54:31.974708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.974547 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\") pod \"ef49a238-073d-4ce2-9f7c-3b1391196bde\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " Oct 30 21:54:31.974708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.974597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\") pod \"ef49a238-073d-4ce2-9f7c-3b1391196bde\" (UID: \"ef49a238-073d-4ce2-9f7c-3b1391196bde\") " Oct 30 21:54:31.976056 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.976018 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca" pod "ef49a238-073d-4ce2-9f7c-3b1391196bde" (UID: "ef49a238-073d-4ce2-9f7c-3b1391196bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:31.977190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.977159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca" pod "ef49a238-073d-4ce2-9f7c-3b1391196bde" (UID: "ef49a238-073d-4ce2-9f7c-3b1391196bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:31.977368 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:31.977343 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq" pod "ef49a238-073d-4ce2-9f7c-3b1391196bde" (UID: "ef49a238-073d-4ce2-9f7c-3b1391196bde"). InnerVolumeSpecName "kube-api-access-x97hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:32.077350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.077308 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.077350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.077351 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef49a238-073d-4ce2-9f7c-3b1391196bde-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.077607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.077368 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97hq\" (UniqueName: \"kubernetes.io/projected/ef49a238-073d-4ce2-9f7c-3b1391196bde-kube-api-access-x97hq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.733933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.733901 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" Oct 30 21:54:32.734437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.733900 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp" event={"ID":"ef49a238-073d-4ce2-9f7c-3b1391196bde","Type":"ContainerDied","Data":"a42cc7da4b990a452847ee2683efd9cf3b0b9d3f379c23734f42334babddf16a"} Oct 30 21:54:32.734437 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.734028 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42cc7da4b990a452847ee2683efd9cf3b0b9d3f379c23734f42334babddf16a" Oct 30 21:54:32.750628 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.750600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:32.751116 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.751083 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:32.752189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:32.752165 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 21:54:36.413200 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:36.413135 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:54:36.448034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:36.448000 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:54:38.034662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.034627 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq"] Oct 30 21:54:38.035446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035425 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f3efb4f7-a766-4a73-af33-2fc5903ddc55" containerName="revoke-keys" Oct 30 21:54:38.035446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035444 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3efb4f7-a766-4a73-af33-2fc5903ddc55" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035465 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ef49a238-073d-4ce2-9f7c-3b1391196bde" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035471 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef49a238-073d-4ce2-9f7c-3b1391196bde" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035491 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035496 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035594 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3efb4f7-a766-4a73-af33-2fc5903ddc55" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035605 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef49a238-073d-4ce2-9f7c-3b1391196bde" containerName="revoke-keys" Oct 30 21:54:38.035629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.035614 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" containerName="revoke-keys" Oct 30 21:54:38.040738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.040717 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.050421 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.050323 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:38.052153 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.052124 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:38.052317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.052155 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:38.052317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.052204 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:38.059465 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.059442 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq"] Oct 30 21:54:38.175022 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.174985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.175205 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.175186 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.175341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.175325 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fz2g\" (UniqueName: \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.276366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.276331 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.276583 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.276458 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fz2g\" (UniqueName: \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.276583 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.276560 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.277349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.277328 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.279434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.279410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.294307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.294208 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fz2g\" (UniqueName: \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\") pod \"maas-api-revoke-sub-07f33970-4mnqq\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.328428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.328395 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc"] Oct 30 21:54:38.334108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.334090 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.345202 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.343829 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc"] Oct 30 21:54:38.362372 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.362340 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:38.479350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.479316 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.479524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.479368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqps\" (UniqueName: \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.479626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.479600 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.580609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.580523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.580609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.580575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqps\" (UniqueName: \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.580786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.580685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.581398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.581335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.583640 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.583607 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.596282 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.596241 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqps\" (UniqueName: \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\") pod \"maas-api-revoke-sub-8d6a354b-2qqmc\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.658286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.656474 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:38.718290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.716157 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq"] Oct 30 21:54:38.803955 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:38.803811 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" event={"ID":"aa9f54d1-e72d-431d-af6c-db7f63e13534","Type":"ContainerStarted","Data":"342eef472dc8a529e894f111e6fb6b0b73764acc0e26188af8a2004f3cd8e061"} Oct 30 21:54:39.573611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:39.563060 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc"] Oct 30 21:54:39.937314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:39.932973 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" event={"ID":"aa9f54d1-e72d-431d-af6c-db7f63e13534","Type":"ContainerStarted","Data":"e1312d1c01aa21cf9a1c18b98215ebae57a103194b5cef28f3d30dcd87672f1a"} Oct 30 21:54:40.018341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.012237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" event={"ID":"264c4133-0568-4128-b34f-c1db323b012a","Type":"ContainerStarted","Data":"5d717a91d20a735ebbbd09db495527d56a3216d9ecb88a164c8c9e77d94d07b7"} Oct 30 21:54:40.018341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.016758 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" podStartSLOduration=2.016739716 podStartE2EDuration="2.016739716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:39.995638252 +0000 UTC m=+903.945593202" watchObservedRunningTime="2026-10-30 21:54:40.016739716 +0000 UTC m=+903.966694662" Oct 30 21:54:40.126302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.122567 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.300339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.295370 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.309315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.306349 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.324289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.316037 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.748290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.747533 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.828292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:40.811163 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:54:41.106408 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.089544 5036 generic.go:356] "Generic (PLEG): container finished" podID="264c4133-0568-4128-b34f-c1db323b012a" containerID="e4a980497a93b9958aa774372917a512f8e1df9313e67cad8c209dd6f36d0bd3" exitCode=0 Oct 30 21:54:41.106408 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.089742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" event={"ID":"264c4133-0568-4128-b34f-c1db323b012a","Type":"ContainerDied","Data":"e4a980497a93b9958aa774372917a512f8e1df9313e67cad8c209dd6f36d0bd3"} Oct 30 21:54:41.124293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.121309 5036 generic.go:356] "Generic (PLEG): container finished" podID="aa9f54d1-e72d-431d-af6c-db7f63e13534" containerID="e1312d1c01aa21cf9a1c18b98215ebae57a103194b5cef28f3d30dcd87672f1a" exitCode=0 Oct 30 21:54:41.124293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.121425 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" event={"ID":"aa9f54d1-e72d-431d-af6c-db7f63e13534","Type":"ContainerDied","Data":"e1312d1c01aa21cf9a1c18b98215ebae57a103194b5cef28f3d30dcd87672f1a"} Oct 30 21:54:41.208330 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.204351 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.278531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.277351 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.298299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.297347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.298299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.298078 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:41.328304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.308338 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.328304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.309043 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:41.328304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.316790 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:54:41.328304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.317145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:41.907337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.820427 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.907337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:41.821176 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:42.529220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.529172 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:54:42.536604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.536563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 21:54:42.570291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.559411 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:42.688951 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.629491 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:54:42.821747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.821718 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:42.822374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.822355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:42.822575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.822563 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:42.954316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:42.892325 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:43.133828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.133590 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:43.137294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.137063 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:43.234919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.234893 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:43.279072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.279045 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:43.327570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.327541 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\") pod \"aa9f54d1-e72d-431d-af6c-db7f63e13534\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " Oct 30 21:54:43.327734 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.327608 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\" (UniqueName: \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\") pod \"aa9f54d1-e72d-431d-af6c-db7f63e13534\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " Oct 30 21:54:43.327778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.327752 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\") pod \"aa9f54d1-e72d-431d-af6c-db7f63e13534\" (UID: \"aa9f54d1-e72d-431d-af6c-db7f63e13534\") " Oct 30 21:54:43.328014 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.327983 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca" pod "aa9f54d1-e72d-431d-af6c-db7f63e13534" (UID: "aa9f54d1-e72d-431d-af6c-db7f63e13534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:43.328743 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.328609 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.330052 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.330026 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca" pod "aa9f54d1-e72d-431d-af6c-db7f63e13534" (UID: "aa9f54d1-e72d-431d-af6c-db7f63e13534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:43.330165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.330074 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g" pod "aa9f54d1-e72d-431d-af6c-db7f63e13534" (UID: "aa9f54d1-e72d-431d-af6c-db7f63e13534"). InnerVolumeSpecName "kube-api-access-5fz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:43.429487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.429405 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\") pod \"264c4133-0568-4128-b34f-c1db323b012a\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " Oct 30 21:54:43.429633 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.429509 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\" (UniqueName: \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\") pod \"264c4133-0568-4128-b34f-c1db323b012a\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " Oct 30 21:54:43.429633 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.429563 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\") pod \"264c4133-0568-4128-b34f-c1db323b012a\" (UID: \"264c4133-0568-4128-b34f-c1db323b012a\") " Oct 30 21:54:43.429869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.429831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca" pod "264c4133-0568-4128-b34f-c1db323b012a" (UID: "264c4133-0568-4128-b34f-c1db323b012a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:43.430231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.430203 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fz2g\" (UniqueName: \"kubernetes.io/projected/aa9f54d1-e72d-431d-af6c-db7f63e13534-kube-api-access-5fz2g\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.430328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.430234 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f54d1-e72d-431d-af6c-db7f63e13534-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.430328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.430249 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264c4133-0568-4128-b34f-c1db323b012a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.432144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.432107 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps" pod "264c4133-0568-4128-b34f-c1db323b012a" (UID: "264c4133-0568-4128-b34f-c1db323b012a"). InnerVolumeSpecName "kube-api-access-dxqps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:43.432307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.432254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca" pod "264c4133-0568-4128-b34f-c1db323b012a" (UID: "264c4133-0568-4128-b34f-c1db323b012a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:43.532320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.532284 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqps\" (UniqueName: \"kubernetes.io/projected/264c4133-0568-4128-b34f-c1db323b012a-kube-api-access-dxqps\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.532495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.532325 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264c4133-0568-4128-b34f-c1db323b012a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.547120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.547086 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9"] Oct 30 21:54:43.547992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.547969 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aa9f54d1-e72d-431d-af6c-db7f63e13534" containerName="revoke-keys" Oct 30 21:54:43.547992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.547993 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9f54d1-e72d-431d-af6c-db7f63e13534" containerName="revoke-keys" Oct 30 21:54:43.548131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.548014 5036 cpu_manager.go:414] "containerMap: removing container" podUID="264c4133-0568-4128-b34f-c1db323b012a" containerName="revoke-keys" Oct 30 21:54:43.548131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.548020 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264c4133-0568-4128-b34f-c1db323b012a" containerName="revoke-keys" Oct 30 21:54:43.548236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.548162 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9f54d1-e72d-431d-af6c-db7f63e13534" containerName="revoke-keys" Oct 30 21:54:43.548236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.548174 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="264c4133-0568-4128-b34f-c1db323b012a" containerName="revoke-keys" Oct 30 21:54:43.552325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.552305 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.564825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.564797 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9"] Oct 30 21:54:43.634037 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.634005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.634242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.634140 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbls\" (UniqueName: \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.634376 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.634288 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.736136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.736043 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbls\" (UniqueName: \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.736341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.736161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.736416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.736354 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.737128 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.737084 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.738852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.738828 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.749814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.749782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbls\" (UniqueName: \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\") pod \"maas-api-revoke-sub-e92926d6-dc6v9\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:43.751213 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.751175 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" event={"ID":"264c4133-0568-4128-b34f-c1db323b012a","Type":"ContainerDied","Data":"5d717a91d20a735ebbbd09db495527d56a3216d9ecb88a164c8c9e77d94d07b7"} Oct 30 21:54:43.751353 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.751233 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d717a91d20a735ebbbd09db495527d56a3216d9ecb88a164c8c9e77d94d07b7" Oct 30 21:54:43.751353 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.751319 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc" Oct 30 21:54:43.755486 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.755424 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" event={"ID":"aa9f54d1-e72d-431d-af6c-db7f63e13534","Type":"ContainerDied","Data":"342eef472dc8a529e894f111e6fb6b0b73764acc0e26188af8a2004f3cd8e061"} Oct 30 21:54:43.755645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.755622 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342eef472dc8a529e894f111e6fb6b0b73764acc0e26188af8a2004f3cd8e061" Oct 30 21:54:43.755957 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.755931 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq" Oct 30 21:54:43.882436 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:43.882402 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:44.186877 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:54:44.186837 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842439b9_b906_46bc_8179_38dd22461bed.slice/crio-8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463 WatchSource:0}: Error finding container 8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463: Status 404 returned error can't find the container with id 8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463 Oct 30 21:54:44.192893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:44.192869 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9"] Oct 30 21:54:44.764225 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:44.764176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" event={"ID":"842439b9-b906-46bc-8179-38dd22461bed","Type":"ContainerStarted","Data":"70d14df59e8803074bbb784ef2dee9f8f95fb8ef35391490396cf025d2380ce0"} Oct 30 21:54:44.764409 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:44.764245 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" event={"ID":"842439b9-b906-46bc-8179-38dd22461bed","Type":"ContainerStarted","Data":"8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463"} Oct 30 21:54:44.813313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:44.813192 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" podStartSLOduration=1.81317284 podStartE2EDuration="1.81317284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:44.805355214 +0000 UTC m=+908.755310178" watchObservedRunningTime="2026-10-30 21:54:44.81317284 +0000 UTC m=+908.763127795" Oct 30 21:54:45.774446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:45.774408 5036 generic.go:356] "Generic (PLEG): container finished" podID="842439b9-b906-46bc-8179-38dd22461bed" containerID="70d14df59e8803074bbb784ef2dee9f8f95fb8ef35391490396cf025d2380ce0" exitCode=0 Oct 30 21:54:45.774925 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:45.774552 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" event={"ID":"842439b9-b906-46bc-8179-38dd22461bed","Type":"ContainerDied","Data":"70d14df59e8803074bbb784ef2dee9f8f95fb8ef35391490396cf025d2380ce0"} Oct 30 21:54:46.945561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.945532 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:46.982195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.982163 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\" (UniqueName: \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\") pod \"842439b9-b906-46bc-8179-38dd22461bed\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " Oct 30 21:54:46.982389 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.982311 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\") pod \"842439b9-b906-46bc-8179-38dd22461bed\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " Oct 30 21:54:46.982573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.982549 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\") pod \"842439b9-b906-46bc-8179-38dd22461bed\" (UID: \"842439b9-b906-46bc-8179-38dd22461bed\") " Oct 30 21:54:46.982894 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.982868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca" pod "842439b9-b906-46bc-8179-38dd22461bed" (UID: "842439b9-b906-46bc-8179-38dd22461bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:46.983116 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.983099 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842439b9-b906-46bc-8179-38dd22461bed-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.985098 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.985072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca" pod "842439b9-b906-46bc-8179-38dd22461bed" (UID: "842439b9-b906-46bc-8179-38dd22461bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:46.985199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:46.985101 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls" pod "842439b9-b906-46bc-8179-38dd22461bed" (UID: "842439b9-b906-46bc-8179-38dd22461bed"). InnerVolumeSpecName "kube-api-access-dmbls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:47.083954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:47.083878 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842439b9-b906-46bc-8179-38dd22461bed-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:47.083954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:47.083905 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbls\" (UniqueName: \"kubernetes.io/projected/842439b9-b906-46bc-8179-38dd22461bed-kube-api-access-dmbls\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:47.793689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:47.793662 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" Oct 30 21:54:47.793689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:47.793686 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9" event={"ID":"842439b9-b906-46bc-8179-38dd22461bed","Type":"ContainerDied","Data":"8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463"} Oct 30 21:54:47.793901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:47.793734 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a20e63d882f2197b018fa850d62c514e79877bf3a8fc98a41c1af37ff796463" Oct 30 21:54:48.099632 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:54:48.099552 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:54:51.806291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:51.805373 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.840289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:51.839736 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.926290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:51.925815 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:54:52.073355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.072430 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.155291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.147021 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:54:52.226291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.224349 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.554403 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.554349 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.825355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.816445 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.825355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.816586 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:52.852562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.852517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:54:52.901298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.854410 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:53.115383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:52.948439 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.115383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.094587 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:53.115383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.097568 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.115383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.097937 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:54:53.210293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.209710 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.224303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.211130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:53.249414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.225352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.249414 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.225985 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:54:53.562539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.561365 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 21:54:53.562539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.562090 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:53.820146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.820032 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.888498 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:53.888427 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.097341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.094206 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.098475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.098426 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.244305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.241206 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 21:54:54.267933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.267359 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.568834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.568785 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.830344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.830224 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:54:54.900410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:54.900362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.104353 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.099352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 21:54:55.138315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.104590 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:54:55.236986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.235602 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:54:55.253450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.253402 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 21:54:55.509290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.491299 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:54:55.581661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.580862 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.585716 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.585501 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:54:55.832038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.831937 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.903401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.903360 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.927611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.927583 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:54:55.951015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:55.950988 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:54:56.174918 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:56.174894 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:54:56.478915 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:56.478830 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:54:59.222883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.222851 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl"] Oct 30 21:54:59.223562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.223545 5036 cpu_manager.go:414] "containerMap: removing container" podUID="842439b9-b906-46bc-8179-38dd22461bed" containerName="revoke-keys" Oct 30 21:54:59.223619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.223563 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842439b9-b906-46bc-8179-38dd22461bed" containerName="revoke-keys" Oct 30 21:54:59.223720 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.223709 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="842439b9-b906-46bc-8179-38dd22461bed" containerName="revoke-keys" Oct 30 21:54:59.227762 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.227743 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.246089 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.246062 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:59.246843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.246821 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:59.265532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.265504 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:59.265532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.265512 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:59.265703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.265538 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl"] Oct 30 21:54:59.366038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.365998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.366221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.366159 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.366362 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.366339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvknr\" (UniqueName: \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.501781 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.501683 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.503361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.503309 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.503494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.503466 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvknr\" (UniqueName: \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.503916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.502692 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.515584 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.515508 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.523849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.523827 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvknr\" (UniqueName: \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\") pod \"maas-api-revoke-sub-dd443a01-4lbgl\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.546305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.546181 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:54:59.873113 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:54:59.873015 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121ab45f_f276_4708_ac06_7f8acb92d05c.slice/crio-166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae WatchSource:0}: Error finding container 166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae: Status 404 returned error can't find the container with id 166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae Oct 30 21:54:59.878868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:54:59.875007 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl"] Oct 30 21:55:00.041676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:00.041645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" event={"ID":"121ab45f-f276-4708-ac06-7f8acb92d05c","Type":"ContainerStarted","Data":"166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae"} Oct 30 21:55:01.053995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.053900 5036 generic.go:356] "Generic (PLEG): container finished" podID="121ab45f-f276-4708-ac06-7f8acb92d05c" containerID="638c7393a8ab11379efb2dc07a9cbdd4eb124d8385214f117291429e0a8f384e" exitCode=0 Oct 30 21:55:01.054530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.054061 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" event={"ID":"121ab45f-f276-4708-ac06-7f8acb92d05c","Type":"ContainerDied","Data":"638c7393a8ab11379efb2dc07a9cbdd4eb124d8385214f117291429e0a8f384e"} Oct 30 21:55:01.273361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.273324 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:55:01.277756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.277738 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.297431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.297408 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-689a2563-dockercfg-hj5b8\"" Oct 30 21:55:01.297561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.297519 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-689a2563\"" Oct 30 21:55:01.341401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.341302 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:55:01.346542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.346514 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.349833 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.349808 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:55:01.380509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.380474 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:55:01.444342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.436806 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.444342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.436897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7l65\" (UniqueName: \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.444342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.436989 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4j9w\" (UniqueName: \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.444342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.437020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.540836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.538488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7l65\" (UniqueName: \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.540836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.538609 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4j9w\" (UniqueName: \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.540836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.538639 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.540836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.538865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.543571 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.542141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.543571 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.543515 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.555757 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.555710 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7l65\" (UniqueName: \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-gpjdp\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.561767 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.561734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4j9w\" (UniqueName: \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:01.591989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.591910 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:01.709320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:01.709256 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:02.017444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.003280 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:55:02.019544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.019497 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:55:02.084419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.084336 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" event={"ID":"f3e94876-1910-4bec-85ce-33d49517cad0","Type":"ContainerStarted","Data":"f374863ce6b1f99ee812a5f0b4489f11065f9ef937f9dc18835317722ba0fb4b"} Oct 30 21:55:02.086712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.086641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" event={"ID":"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06","Type":"ContainerStarted","Data":"04b0f52c597a0bf7e4ce5d22bce2d87ab4cc1b224a4974e0123f49867d813c5e"} Oct 30 21:55:02.261431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.261403 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:55:02.351844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.351805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\") pod \"121ab45f-f276-4708-ac06-7f8acb92d05c\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " Oct 30 21:55:02.352042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.351864 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\") pod \"121ab45f-f276-4708-ac06-7f8acb92d05c\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " Oct 30 21:55:02.352042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.351901 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\" (UniqueName: \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\") pod \"121ab45f-f276-4708-ac06-7f8acb92d05c\" (UID: \"121ab45f-f276-4708-ac06-7f8acb92d05c\") " Oct 30 21:55:02.352645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.352296 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca" pod "121ab45f-f276-4708-ac06-7f8acb92d05c" (UID: "121ab45f-f276-4708-ac06-7f8acb92d05c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:02.353168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.353123 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:02.354819 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.354791 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca" pod "121ab45f-f276-4708-ac06-7f8acb92d05c" (UID: "121ab45f-f276-4708-ac06-7f8acb92d05c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:02.355617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.355592 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr" pod "121ab45f-f276-4708-ac06-7f8acb92d05c" (UID: "121ab45f-f276-4708-ac06-7f8acb92d05c"). InnerVolumeSpecName "kube-api-access-rvknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:02.454067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.454031 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ab45f-f276-4708-ac06-7f8acb92d05c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:02.454067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:02.454064 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvknr\" (UniqueName: \"kubernetes.io/projected/121ab45f-f276-4708-ac06-7f8acb92d05c-kube-api-access-rvknr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.102911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.102870 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" Oct 30 21:55:03.102911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.102895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl" event={"ID":"121ab45f-f276-4708-ac06-7f8acb92d05c","Type":"ContainerDied","Data":"166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae"} Oct 30 21:55:03.103490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.102945 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="166b88b542e5bc0c50b5ac9db1caf95bc82aff04549eefdbe9d4cbff8bd143ae" Oct 30 21:55:03.105344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.105301 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" event={"ID":"f3e94876-1910-4bec-85ce-33d49517cad0","Type":"ContainerStarted","Data":"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385"} Oct 30 21:55:03.105484 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.105408 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:03.107858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.107834 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" event={"ID":"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06","Type":"ContainerStarted","Data":"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e"} Oct 30 21:55:03.108318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.108296 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:03.162163 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.162107 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" podStartSLOduration=2.162091104 podStartE2EDuration="2.162091104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:03.160991735 +0000 UTC m=+927.110946682" watchObservedRunningTime="2026-10-30 21:55:03.162091104 +0000 UTC m=+927.112046050" Oct 30 21:55:03.162418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:03.162225 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" podStartSLOduration=2.162214921 podStartE2EDuration="2.162214921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:03.129495027 +0000 UTC m=+927.079449972" watchObservedRunningTime="2026-10-30 21:55:03.162214921 +0000 UTC m=+927.112169871" Oct 30 21:55:07.553078 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.553042 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb"] Oct 30 21:55:07.554043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.554018 5036 cpu_manager.go:414] "containerMap: removing container" podUID="121ab45f-f276-4708-ac06-7f8acb92d05c" containerName="revoke-keys" Oct 30 21:55:07.554043 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.554041 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121ab45f-f276-4708-ac06-7f8acb92d05c" containerName="revoke-keys" Oct 30 21:55:07.554234 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.554221 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="121ab45f-f276-4708-ac06-7f8acb92d05c" containerName="revoke-keys" Oct 30 21:55:07.563807 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.563781 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.566452 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.566425 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb"] Oct 30 21:55:07.567635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.567615 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:07.568148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.568101 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:07.568148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.568106 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:55:07.568148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.568105 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:07.733329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.733284 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.733540 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.733361 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxbt\" (UniqueName: \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.733611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.733536 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.834870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.834773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.834870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.834858 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxbt\" (UniqueName: \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.835061 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.834931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.835630 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.835599 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.837680 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.837651 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.846656 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.846625 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxbt\" (UniqueName: \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\") pod \"maas-api-revoke-sub-208470cf-tg5gb\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:07.883697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:07.883668 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:08.224145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:08.224113 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb"] Oct 30 21:55:09.217536 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:09.217505 5036 generic.go:356] "Generic (PLEG): container finished" podID="ab7fefd0-b8d8-4ab3-8610-a45d61897334" containerID="4e8403af1a25f0384aa381b0a4ba779dd6a604cb91f72cc4d09090672206a330" exitCode=0 Oct 30 21:55:09.217942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:09.217591 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" event={"ID":"ab7fefd0-b8d8-4ab3-8610-a45d61897334","Type":"ContainerDied","Data":"4e8403af1a25f0384aa381b0a4ba779dd6a604cb91f72cc4d09090672206a330"} Oct 30 21:55:09.217942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:09.217655 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" event={"ID":"ab7fefd0-b8d8-4ab3-8610-a45d61897334","Type":"ContainerStarted","Data":"3c04d7c0875d17ca5e058bb7fae16de798181de84a2ad4c13281f6ac42c93ca6"} Oct 30 21:55:10.437976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.437943 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:10.574365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.574260 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\") pod \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " Oct 30 21:55:10.574531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.574377 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\" (UniqueName: \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\") pod \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " Oct 30 21:55:10.574531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.574507 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\") pod \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\" (UID: \"ab7fefd0-b8d8-4ab3-8610-a45d61897334\") " Oct 30 21:55:10.574683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.574656 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca" pod "ab7fefd0-b8d8-4ab3-8610-a45d61897334" (UID: "ab7fefd0-b8d8-4ab3-8610-a45d61897334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:10.574953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.574938 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:10.576619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.576599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt" pod "ab7fefd0-b8d8-4ab3-8610-a45d61897334" (UID: "ab7fefd0-b8d8-4ab3-8610-a45d61897334"). InnerVolumeSpecName "kube-api-access-kvxbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:10.576835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.576815 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca" pod "ab7fefd0-b8d8-4ab3-8610-a45d61897334" (UID: "ab7fefd0-b8d8-4ab3-8610-a45d61897334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:10.676731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.676696 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxbt\" (UniqueName: \"kubernetes.io/projected/ab7fefd0-b8d8-4ab3-8610-a45d61897334-kube-api-access-kvxbt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:10.676731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:10.676724 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7fefd0-b8d8-4ab3-8610-a45d61897334-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:11.050634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.050599 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 21:55:11.051633 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.051614 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ab7fefd0-b8d8-4ab3-8610-a45d61897334" containerName="revoke-keys" Oct 30 21:55:11.051700 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.051637 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7fefd0-b8d8-4ab3-8610-a45d61897334" containerName="revoke-keys" Oct 30 21:55:11.051830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.051817 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7fefd0-b8d8-4ab3-8610-a45d61897334" containerName="revoke-keys" Oct 30 21:55:11.057187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.057164 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.075664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.075634 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 21:55:11.186246 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186202 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186457 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186324 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186457 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186444 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186506 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186580 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dz8g\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186713 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186749 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186836 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.186932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.186910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.242477 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.242436 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" event={"ID":"ab7fefd0-b8d8-4ab3-8610-a45d61897334","Type":"ContainerDied","Data":"3c04d7c0875d17ca5e058bb7fae16de798181de84a2ad4c13281f6ac42c93ca6"} Oct 30 21:55:11.242680 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.242488 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c04d7c0875d17ca5e058bb7fae16de798181de84a2ad4c13281f6ac42c93ca6" Oct 30 21:55:11.242680 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.242490 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb" Oct 30 21:55:11.287916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.287883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dz8g\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.287959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.287978 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288025 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288071 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288142 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288351 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288451 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288459 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.288695 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.288661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.289327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.289301 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.290766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.290746 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.290902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.290884 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.300631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.300608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.300862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.300816 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dz8g\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.377218 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.377174 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:11.671416 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:55:11.671380 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20764b52_2d02_4d45_b5cb_722bb0e8e2a6.slice/crio-047f98304daf53e81eddaf767f5a310f383f8448bc122a4dab960db767d6638e WatchSource:0}: Error finding container 047f98304daf53e81eddaf767f5a310f383f8448bc122a4dab960db767d6638e: Status 404 returned error can't find the container with id 047f98304daf53e81eddaf767f5a310f383f8448bc122a4dab960db767d6638e Oct 30 21:55:11.672015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.671994 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 21:55:11.673969 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.673931 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:11.674076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.674031 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:11.674136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:11.674083 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:12.258364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.258325 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" event={"ID":"20764b52-2d02-4d45-b5cb-722bb0e8e2a6","Type":"ContainerStarted","Data":"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7"} Oct 30 21:55:12.258589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.258390 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" event={"ID":"20764b52-2d02-4d45-b5cb-722bb0e8e2a6","Type":"ContainerStarted","Data":"047f98304daf53e81eddaf767f5a310f383f8448bc122a4dab960db767d6638e"} Oct 30 21:55:12.283390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.283334 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podStartSLOduration=2.283320332 podStartE2EDuration="2.283320332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:12.280765219 +0000 UTC m=+936.230720164" watchObservedRunningTime="2026-10-30 21:55:12.283320332 +0000 UTC m=+936.233275275" Oct 30 21:55:12.381648 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.381617 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:12.384475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.384444 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:12.384696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:12.384669 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:13.575229 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:13.575193 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:13.575724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:13.575253 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:14.105903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.105856 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.155291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.151366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.176299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.175438 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:55:14.254141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.254109 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:55:14.472346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.380642 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:14.472346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.380695 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:14.559296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.549375 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.595518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.595466 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.832347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.820358 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.880311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.868254 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.918015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:14.906569 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.138516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.136160 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:15.138516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.136713 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.154211 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.152729 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.155960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.154591 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.156730 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.156714 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:55:15.156987 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.156976 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:55:15.581584 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.549551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.642785 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.635532 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:55:15.677324 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.672561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 21:55:15.677324 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.673793 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:55:15.863312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.842957 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:55:15.863312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.844346 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:55:15.888290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.874791 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.888290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.875777 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:55:15.915188 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.904290 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:15.915188 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.904372 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:15.981503 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:15.981460 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.144676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.144646 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:55:16.427390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.310963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.459323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.455282 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:16.459323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.455353 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:16.459323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.457160 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.551702 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.551648 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.629344 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.629294 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.638973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.638938 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:16.639327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.639294 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.683812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.683711 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.747293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.703195 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.925490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.845392 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.938353 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.925758 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.990548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:16.970751 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:17.154287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.151341 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:17.331345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.311950 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:55:17.521343 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:55:17.327425 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.108s" Oct 30 21:55:17.521618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.521597 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:55:17.555157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.555125 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:17.555449 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.555419 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:17.630336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.626427 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 21:55:17.830311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.804533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 21:55:17.894350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.856799 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:55:17.968292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:17.939401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.551066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.434347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:55:18.551066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.524954 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:18.551066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.525014 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:18.680426 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.680361 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.747294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.729425 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:55:18.854491 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.854383 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:18.880434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.880363 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.895537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.895489 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.964466 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.929222 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 21:55:18.968648 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.939539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 21:55:18.982510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.975541 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-v8kmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:18.982510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:18.975608 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.995300 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:55:18.988349 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:55:19.185563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.149151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:19.196420 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.194938 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8"] Oct 30 21:55:19.214939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.212664 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 21:55:19.214939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.212700 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8"] Oct 30 21:55:19.214939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.212758 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 21:55:19.214939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.212788 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:55:19.214939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.213473 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.217157 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.217140 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 21:55:19.221825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.221806 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:19.222051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.222036 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:19.224484 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.224467 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:19.231410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.231391 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:55:19.242321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.237776 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:55:19.377782 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.377572 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:19.377782 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.377622 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:19.399189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.399149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.399356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.399334 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.399432 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.399359 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9sr\" (UniqueName: \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.500604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.500410 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.500604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.500442 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9sr\" (UniqueName: \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.500848 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.500641 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.501489 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.501466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.507127 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.507092 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.569172 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.569114 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9sr\" (UniqueName: \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\") pod \"maas-api-revoke-sub-51d72c72-6hmq8\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:19.874527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:19.874446 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:20.377431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:20.377405 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8"] Oct 30 21:55:20.377775 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:20.377750 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:20.377897 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:20.377795 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:21.251222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:21.251187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" event={"ID":"0779815a-7935-4123-9a31-59d68b2abd67","Type":"ContainerStarted","Data":"4aeb3f4cd10649b94741a12a662be99469e820c9bbdf8d27ad1514ab0cc23082"} Oct 30 21:55:21.252369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:21.252342 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" event={"ID":"0779815a-7935-4123-9a31-59d68b2abd67","Type":"ContainerStarted","Data":"65813d614adf92206518c8b1a22feea373b107def64463521f74e3db83121bc6"} Oct 30 21:55:21.270950 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:21.270900 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" podStartSLOduration=4.27088484 podStartE2EDuration="4.27088484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:21.26933445 +0000 UTC m=+945.219289396" watchObservedRunningTime="2026-10-30 21:55:21.27088484 +0000 UTC m=+945.220839785" Oct 30 21:55:21.378413 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:21.378385 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:55:21.378577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:21.378437 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:55:22.261561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.261526 5036 generic.go:356] "Generic (PLEG): container finished" podID="0779815a-7935-4123-9a31-59d68b2abd67" containerID="4aeb3f4cd10649b94741a12a662be99469e820c9bbdf8d27ad1514ab0cc23082" exitCode=0 Oct 30 21:55:22.261923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.261610 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" event={"ID":"0779815a-7935-4123-9a31-59d68b2abd67","Type":"ContainerDied","Data":"4aeb3f4cd10649b94741a12a662be99469e820c9bbdf8d27ad1514ab0cc23082"} Oct 30 21:55:22.382777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.382738 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:22.383302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.383282 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:22.384025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.384002 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 21:55:22.977961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.977921 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 21:55:22.982783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.982762 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:22.999566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:22.996472 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 21:55:23.113872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.113836 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.113899 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pgh\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114130 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114168 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114199 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114362 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114239 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114362 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114302 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114358 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.114451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.114425 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215536 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215499 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215536 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215535 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215571 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215587 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215602 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215628 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.215772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215768 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.215815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pgh\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216072 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.216024 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.216094 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.216284 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.216338 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.216570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.216551 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.219046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.219017 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.219520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.219499 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.228179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.228121 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pgh\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.229007 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.228975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.310984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.310949 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:23.469152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.469116 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:23.622316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.622209 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\" (UniqueName: \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\") pod \"0779815a-7935-4123-9a31-59d68b2abd67\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " Oct 30 21:55:23.622481 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.622383 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\") pod \"0779815a-7935-4123-9a31-59d68b2abd67\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " Oct 30 21:55:23.622481 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.622423 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\") pod \"0779815a-7935-4123-9a31-59d68b2abd67\" (UID: \"0779815a-7935-4123-9a31-59d68b2abd67\") " Oct 30 21:55:23.623245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.623216 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca" pod "0779815a-7935-4123-9a31-59d68b2abd67" (UID: "0779815a-7935-4123-9a31-59d68b2abd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:23.624813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.624786 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr" pod "0779815a-7935-4123-9a31-59d68b2abd67" (UID: "0779815a-7935-4123-9a31-59d68b2abd67"). InnerVolumeSpecName "kube-api-access-vl9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:23.625114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.625092 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca" pod "0779815a-7935-4123-9a31-59d68b2abd67" (UID: "0779815a-7935-4123-9a31-59d68b2abd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:23.646171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.646149 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 21:55:23.646307 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:55:23.646290 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2445cf25_888e_4836_971e_ac2ecd90eea5.slice/crio-b76570a38ec4faf95b8f19be269706e3ab29e0a8ea456970862a40b9cf6c9f11 WatchSource:0}: Error finding container b76570a38ec4faf95b8f19be269706e3ab29e0a8ea456970862a40b9cf6c9f11: Status 404 returned error can't find the container with id b76570a38ec4faf95b8f19be269706e3ab29e0a8ea456970862a40b9cf6c9f11 Oct 30 21:55:23.648845 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.648807 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:23.648926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.648877 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:23.648926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.648911 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:23.724203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.724180 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9sr\" (UniqueName: \"kubernetes.io/projected/0779815a-7935-4123-9a31-59d68b2abd67-kube-api-access-vl9sr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:23.724318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.724207 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:23.724318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:23.724218 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0779815a-7935-4123-9a31-59d68b2abd67-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.281305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.281240 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" event={"ID":"2445cf25-888e-4836-971e-ac2ecd90eea5","Type":"ContainerStarted","Data":"33bb390a5dcea96ca16bc13a527fd61c7c2791c63fcd2a3173e6d3f882ea97f2"} Oct 30 21:55:24.281456 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.281333 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" event={"ID":"2445cf25-888e-4836-971e-ac2ecd90eea5","Type":"ContainerStarted","Data":"b76570a38ec4faf95b8f19be269706e3ab29e0a8ea456970862a40b9cf6c9f11"} Oct 30 21:55:24.282683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.282658 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" Oct 30 21:55:24.282788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.282695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8" event={"ID":"0779815a-7935-4123-9a31-59d68b2abd67","Type":"ContainerDied","Data":"65813d614adf92206518c8b1a22feea373b107def64463521f74e3db83121bc6"} Oct 30 21:55:24.282788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.282741 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65813d614adf92206518c8b1a22feea373b107def64463521f74e3db83121bc6" Oct 30 21:55:24.307361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.307310 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podStartSLOduration=2.307253636 podStartE2EDuration="2.307253636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:24.304872269 +0000 UTC m=+948.254827255" watchObservedRunningTime="2026-10-30 21:55:24.307253636 +0000 UTC m=+948.257208583" Oct 30 21:55:24.324160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.311751 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:24.324160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.321646 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:24.324160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:24.321702 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:25.316624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:25.316589 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:25.317064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:25.317035 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:26.316366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:26.314579 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:26.316366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:26.314638 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:27.342332 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:27.338581 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:27.342332 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:27.338646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:28.328292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:28.316614 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:28.328292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:28.316689 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:29.335347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:29.335310 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:29.335957 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:29.335922 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:30.312492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:30.312452 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:55:30.314738 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:30.312529 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:55:31.319778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:31.319745 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:31.321045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:31.320967 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:31.322327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:31.322304 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 21:55:36.014230 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.014183 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:55:36.015777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.015746 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0779815a-7935-4123-9a31-59d68b2abd67" containerName="revoke-keys" Oct 30 21:55:36.015777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.015776 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0779815a-7935-4123-9a31-59d68b2abd67" containerName="revoke-keys" Oct 30 21:55:36.016126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.016108 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0779815a-7935-4123-9a31-59d68b2abd67" containerName="revoke-keys" Oct 30 21:55:36.022916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.022895 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.039391 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.039363 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:55:36.061804 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.061776 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c9c95-dockercfg-bhtf2\"" Oct 30 21:55:36.093665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.093639 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.093863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.093841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7xg\" (UniqueName: \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.129401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.129357 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c9c95\"" Oct 30 21:55:36.234417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.234379 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.234900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.234876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7xg\" (UniqueName: \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.249587 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.249552 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c9c95\"" Oct 30 21:55:36.250985 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.250952 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.284357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.279207 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7xg\" (UniqueName: \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\") pod \"payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.349683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.349653 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c9c95-dockercfg-bhtf2\"" Oct 30 21:55:36.357712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.357682 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:36.674175 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:55:36.674140 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42024c8_aed2_42ce_bf64_2c2c5759e0b6.slice/crio-4790c865353335843a49ff44de2ba87a99edbf4786ff2bc6d92926758ac54a2f WatchSource:0}: Error finding container 4790c865353335843a49ff44de2ba87a99edbf4786ff2bc6d92926758ac54a2f: Status 404 returned error can't find the container with id 4790c865353335843a49ff44de2ba87a99edbf4786ff2bc6d92926758ac54a2f Oct 30 21:55:36.674577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:36.674553 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:55:37.538863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:37.538827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" event={"ID":"c42024c8-aed2-42ce-bf64-2c2c5759e0b6","Type":"ContainerStarted","Data":"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913"} Oct 30 21:55:37.539387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:37.538885 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" event={"ID":"c42024c8-aed2-42ce-bf64-2c2c5759e0b6","Type":"ContainerStarted","Data":"4790c865353335843a49ff44de2ba87a99edbf4786ff2bc6d92926758ac54a2f"} Oct 30 21:55:37.539387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:37.538915 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:55:37.575494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:37.575437 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" podStartSLOduration=2.575419591 podStartE2EDuration="2.575419591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:37.573246452 +0000 UTC m=+961.523201398" watchObservedRunningTime="2026-10-30 21:55:37.575419591 +0000 UTC m=+961.525374538" Oct 30 21:55:48.101019 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:55:48.100977 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:55:48.551371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:55:48.551342 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:56:00.921907 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:00.921871 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:56:00.927980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:00.927956 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:00.939844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:00.939816 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-79724b-openshift-default-dockercfg-ls5d9\"" Oct 30 21:56:00.948962 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:00.948932 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:56:01.020978 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.020939 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.020975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021025 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021368 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021335 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021383 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9b9\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021506 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021613 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021596 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.021666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.021634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123599 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9b9\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123746 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123814 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.123995 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.124017 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.124080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.124592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.125663 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.125898 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.126731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.126118 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.128212 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.128188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.131429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.131382 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.144796 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.144741 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.146857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.146762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.155976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.155946 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9b9\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\") pod \"e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.244791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.244697 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:01.573071 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:01.573036 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d948f42_c502_4b96_ae12_d052d17a1c55.slice/crio-ad5a591a7deb2a151b6468c01e2e239d922040962c3f63d2b77191f2b35e3f84 WatchSource:0}: Error finding container ad5a591a7deb2a151b6468c01e2e239d922040962c3f63d2b77191f2b35e3f84: Status 404 returned error can't find the container with id ad5a591a7deb2a151b6468c01e2e239d922040962c3f63d2b77191f2b35e3f84 Oct 30 21:56:01.573555 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.573511 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:56:01.575665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.575633 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:01.575744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.575696 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:01.575744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.575724 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:01.784863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.784823 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerStarted","Data":"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b"} Oct 30 21:56:01.785039 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:01.784879 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerStarted","Data":"ad5a591a7deb2a151b6468c01e2e239d922040962c3f63d2b77191f2b35e3f84"} Oct 30 21:56:02.245847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:02.245816 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:02.251715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:02.251690 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:02.281356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:02.281260 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podStartSLOduration=2.281247428 podStartE2EDuration="2.281247428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:01.81483658 +0000 UTC m=+985.764791528" watchObservedRunningTime="2026-10-30 21:56:02.281247428 +0000 UTC m=+986.231202373" Oct 30 21:56:02.795681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:02.795643 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:02.796739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:02.796705 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:56:11.298612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.298569 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 21:56:11.304530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.304503 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.322770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.322738 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 21:56:11.358027 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.357990 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 21:56:11.362592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.362571 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.372802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.372762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.373019 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.372998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373133 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.373329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373311 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373444 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373558 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373668 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8vq\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.373889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.373877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374018 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374100 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7jv\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374432 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.374579 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374595 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374631 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.374693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374660 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.374693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.374681 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.392901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.392874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 21:56:11.475842 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475808 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475877 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8vq\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.475993 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7jv\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476128 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476172 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476248 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476369 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476414 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476425 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476450 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476546 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.476883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.476879 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.477495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.477495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477228 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.477495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477304 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.477675 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477498 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.477675 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477619 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.477790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.477992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.477967 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.479095 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.479068 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.479209 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.479165 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.479373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.479355 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.479997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.479975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.489932 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.489909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.492525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.492499 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8vq\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.493093 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.493066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.493255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.493233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7jv\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.626696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.626623 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:11.694236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.693903 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:11.978312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.978248 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 21:56:11.979626 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:11.979485 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6939fb_1bd2_40c9_b53a_0d0454dc69ae.slice/crio-0c287a466388f092e9e6cd865c9ac0779c9c9fa49324dfaafe2ce7c59270debd WatchSource:0}: Error finding container 0c287a466388f092e9e6cd865c9ac0779c9c9fa49324dfaafe2ce7c59270debd: Status 404 returned error can't find the container with id 0c287a466388f092e9e6cd865c9ac0779c9c9fa49324dfaafe2ce7c59270debd Oct 30 21:56:11.982686 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.982653 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:11.982760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.982733 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:11.982802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:11.982765 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:12.062304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.062250 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:12.062406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.062348 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:12.062406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.062379 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:12.064541 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.064521 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 21:56:12.963570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.963533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" event={"ID":"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae","Type":"ContainerStarted","Data":"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8"} Oct 30 21:56:12.964130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.964109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" event={"ID":"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae","Type":"ContainerStarted","Data":"0c287a466388f092e9e6cd865c9ac0779c9c9fa49324dfaafe2ce7c59270debd"} Oct 30 21:56:12.966665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.966641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" event={"ID":"59e4a593-16fc-4734-a2c2-e3771683cdc5","Type":"ContainerStarted","Data":"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f"} Oct 30 21:56:12.966843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.966830 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" event={"ID":"59e4a593-16fc-4734-a2c2-e3771683cdc5","Type":"ContainerStarted","Data":"22cc799cee6a76eda530d9857c0d394bf1051101a57a1d6d1b3099a707cc0d23"} Oct 30 21:56:12.998781 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:12.998700 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podStartSLOduration=1.9986812349999998 podStartE2EDuration="1.998681235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:12.996685701 +0000 UTC m=+996.946640662" watchObservedRunningTime="2026-10-30 21:56:12.998681235 +0000 UTC m=+996.948636181" Oct 30 21:56:13.631322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.629547 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:13.631322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.629679 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:13.631322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.629735 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:13.695365 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.695333 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:13.705289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.699982 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:13.705289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:13.700052 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:14.639930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:14.639895 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:14.640714 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:14.640681 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:14.696323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:14.695735 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:14.696323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:14.695800 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:15.641406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.641368 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:15.641965 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.641932 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:15.695961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.695925 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:15.696186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.696165 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:15.743286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.739651 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podStartSLOduration=4.739629659 podStartE2EDuration="4.739629659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:13.027581293 +0000 UTC m=+996.977536243" watchObservedRunningTime="2026-10-30 21:56:15.739629659 +0000 UTC m=+999.689584605" Oct 30 21:56:15.759176 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.755494 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9"] Oct 30 21:56:15.782510 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.782477 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9"] Oct 30 21:56:15.782845 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.782827 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.789235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.789115 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:15.789410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.789335 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:15.791108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.790382 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:56:15.791108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.790440 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:15.875289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.874181 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.875289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.874363 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.875289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.874408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.976378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.976220 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.976534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.976499 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.976577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.976558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.977204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.977181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:15.979699 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:15.979680 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:16.001717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.001697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-rlvw9\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:16.107099 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.107068 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:16.590174 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.590148 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9"] Oct 30 21:56:16.633473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.633428 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:16.633610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.633497 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:16.710295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.705725 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:16.710295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:16.705802 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:17.045300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:17.043052 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" event={"ID":"1489954b-e848-4e34-ad38-c896be27ce36","Type":"ContainerStarted","Data":"0f1ef103d97089bdac4da987ecb9c9d02a2bcd0511a7c61b969e0a0e577db228"} Oct 30 21:56:17.667326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:17.646459 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:17.667326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:17.646532 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:17.700291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:17.695723 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:17.700291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:17.695814 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:18.096300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.095284 5036 generic.go:356] "Generic (PLEG): container finished" podID="1489954b-e848-4e34-ad38-c896be27ce36" containerID="33a3c1763bb5d7460948c9db94db527f32d2eb37cb77e597b222656ca94f18bc" exitCode=0 Oct 30 21:56:18.096300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.095429 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" event={"ID":"1489954b-e848-4e34-ad38-c896be27ce36","Type":"ContainerDied","Data":"33a3c1763bb5d7460948c9db94db527f32d2eb37cb77e597b222656ca94f18bc"} Oct 30 21:56:18.163291 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:56:18.162464 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:56:18.630290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.627600 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:56:18.630290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.627667 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:56:18.695031 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.694983 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:18.695232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:18.695055 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:19.378988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.378912 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:19.470339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.470297 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\") pod \"1489954b-e848-4e34-ad38-c896be27ce36\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " Oct 30 21:56:19.470639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.470605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\") pod \"1489954b-e848-4e34-ad38-c896be27ce36\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " Oct 30 21:56:19.470942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.470923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\") pod \"1489954b-e848-4e34-ad38-c896be27ce36\" (UID: \"1489954b-e848-4e34-ad38-c896be27ce36\") " Oct 30 21:56:19.471340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.471311 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca" pod "1489954b-e848-4e34-ad38-c896be27ce36" (UID: "1489954b-e848-4e34-ad38-c896be27ce36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:19.472145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.472122 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:19.473686 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.473646 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9" pod "1489954b-e848-4e34-ad38-c896be27ce36" (UID: "1489954b-e848-4e34-ad38-c896be27ce36"). InnerVolumeSpecName "kube-api-access-bj8b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:19.473899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.473865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca" pod "1489954b-e848-4e34-ad38-c896be27ce36" (UID: "1489954b-e848-4e34-ad38-c896be27ce36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:19.573496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.573456 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/1489954b-e848-4e34-ad38-c896be27ce36-kube-api-access-bj8b9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:19.573496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.573484 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1489954b-e848-4e34-ad38-c896be27ce36-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:19.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.695697 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:56:19.695916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:19.695766 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:56:20.203407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.203022 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" Oct 30 21:56:20.237999 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.236521 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9" event={"ID":"1489954b-e848-4e34-ad38-c896be27ce36","Type":"ContainerDied","Data":"0f1ef103d97089bdac4da987ecb9c9d02a2bcd0511a7c61b969e0a0e577db228"} Oct 30 21:56:20.237999 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.236575 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1ef103d97089bdac4da987ecb9c9d02a2bcd0511a7c61b969e0a0e577db228" Oct 30 21:56:20.261524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.261491 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:20.263001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.262978 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:20.264997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.264975 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 21:56:20.699848 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.699822 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:20.700342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.700325 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:20.701241 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:20.701225 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 21:56:24.546356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.546319 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw"] Oct 30 21:56:24.547050 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.547033 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1489954b-e848-4e34-ad38-c896be27ce36" containerName="revoke-keys" Oct 30 21:56:24.547094 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.547053 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1489954b-e848-4e34-ad38-c896be27ce36" containerName="revoke-keys" Oct 30 21:56:24.547182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.547173 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1489954b-e848-4e34-ad38-c896be27ce36" containerName="revoke-keys" Oct 30 21:56:24.550635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.550618 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.563497 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.563439 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:24.563497 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.563446 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:24.563687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.563514 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:24.563687 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.563660 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:56:24.568398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.568376 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw"] Oct 30 21:56:24.664904 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.664873 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.664904 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.664913 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.665182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.664930 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcz87\" (UniqueName: \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.766516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.766485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.766703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.766526 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.766703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.766544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcz87\" (UniqueName: \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.767281 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.767245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.769156 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.769135 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.784758 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.784735 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcz87\" (UniqueName: \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\") pod \"maas-api-revoke-sub-2685ea84-n8qvw\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:24.866021 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:24.865947 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:25.180368 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.180339 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw"] Oct 30 21:56:25.181314 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:25.181282 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22514316_a5c7_439e_8a87_b36ca39af526.slice/crio-b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501 WatchSource:0}: Error finding container b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501: Status 404 returned error can't find the container with id b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501 Oct 30 21:56:25.272320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.272283 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" event={"ID":"22514316-a5c7-439e-8a87-b36ca39af526","Type":"ContainerStarted","Data":"b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501"} Oct 30 21:56:25.749523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.749487 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5"] Oct 30 21:56:25.754092 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.754075 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:25.770542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.770518 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5"] Oct 30 21:56:25.900210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.896120 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7h8d\" (UniqueName: \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:25.900210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.899564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:25.900210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:25.899933 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.001082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.001050 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.001233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.001119 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7h8d\" (UniqueName: \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.001233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.001184 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.001798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.001771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.003776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.003754 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.017733 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.017711 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7h8d\" (UniqueName: \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\") pod \"maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.070748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.070718 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:26.211587 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.208027 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:56:26.239210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.239172 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:56:26.294086 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.292144 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" event={"ID":"22514316-a5c7-439e-8a87-b36ca39af526","Type":"ContainerStarted","Data":"469cd67a34b1d5cefe47ed2195abd46b61c6b5029cbbe1a8dca60be06c92ec47"} Oct 30 21:56:26.343023 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.342962 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" podStartSLOduration=2.342941174 podStartE2EDuration="2.342941174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:26.338821091 +0000 UTC m=+1010.288776039" watchObservedRunningTime="2026-10-30 21:56:26.342941174 +0000 UTC m=+1010.292896121" Oct 30 21:56:26.388611 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:26.388571 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5f66d3_163b_4baf_93bf_73cf39fcc7bf.slice/crio-788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8 WatchSource:0}: Error finding container 788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8: Status 404 returned error can't find the container with id 788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8 Oct 30 21:56:26.390076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:26.390045 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5"] Oct 30 21:56:27.306152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:27.306116 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" event={"ID":"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf","Type":"ContainerStarted","Data":"734a8844da6c6b2d3ec003ef66e5cdc315993a67f507314d223d0a5840a58497"} Oct 30 21:56:27.306639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:27.306181 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" event={"ID":"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf","Type":"ContainerStarted","Data":"788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8"} Oct 30 21:56:27.307826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:27.307802 5036 generic.go:356] "Generic (PLEG): container finished" podID="22514316-a5c7-439e-8a87-b36ca39af526" containerID="469cd67a34b1d5cefe47ed2195abd46b61c6b5029cbbe1a8dca60be06c92ec47" exitCode=0 Oct 30 21:56:27.307958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:27.307861 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" event={"ID":"22514316-a5c7-439e-8a87-b36ca39af526","Type":"ContainerDied","Data":"469cd67a34b1d5cefe47ed2195abd46b61c6b5029cbbe1a8dca60be06c92ec47"} Oct 30 21:56:27.326079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:27.326028 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" podStartSLOduration=2.32601268 podStartE2EDuration="2.32601268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:27.324373526 +0000 UTC m=+1011.274328463" watchObservedRunningTime="2026-10-30 21:56:27.32601268 +0000 UTC m=+1011.275967626" Oct 30 21:56:28.326522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.326479 5036 generic.go:356] "Generic (PLEG): container finished" podID="1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" containerID="734a8844da6c6b2d3ec003ef66e5cdc315993a67f507314d223d0a5840a58497" exitCode=0 Oct 30 21:56:28.326992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.326769 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" event={"ID":"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf","Type":"ContainerDied","Data":"734a8844da6c6b2d3ec003ef66e5cdc315993a67f507314d223d0a5840a58497"} Oct 30 21:56:28.430613 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.430580 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj"] Oct 30 21:56:28.445055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.438851 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.445055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.444825 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj"] Oct 30 21:56:28.452422 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.452392 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.452665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.452643 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.453006 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.452985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49kw\" (UniqueName: \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.544797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.544773 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:28.554040 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554015 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\") pod \"22514316-a5c7-439e-8a87-b36ca39af526\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " Oct 30 21:56:28.554175 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\") pod \"22514316-a5c7-439e-8a87-b36ca39af526\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " Oct 30 21:56:28.554296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\" (UniqueName: \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\") pod \"22514316-a5c7-439e-8a87-b36ca39af526\" (UID: \"22514316-a5c7-439e-8a87-b36ca39af526\") " Oct 30 21:56:28.554469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554447 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca" pod "22514316-a5c7-439e-8a87-b36ca39af526" (UID: "22514316-a5c7-439e-8a87-b36ca39af526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:28.554674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554654 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49kw\" (UniqueName: \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.554746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554734 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.554806 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.554794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.555018 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.555003 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:28.555670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.555605 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.556634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.556608 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87" pod "22514316-a5c7-439e-8a87-b36ca39af526" (UID: "22514316-a5c7-439e-8a87-b36ca39af526"). InnerVolumeSpecName "kube-api-access-dcz87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:28.556719 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.556609 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca" pod "22514316-a5c7-439e-8a87-b36ca39af526" (UID: "22514316-a5c7-439e-8a87-b36ca39af526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:28.557486 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.557466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.569934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.569905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49kw\" (UniqueName: \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\") pod \"maas-api-revoke-sub-be591507-zlhmj\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:28.656959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.656928 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22514316-a5c7-439e-8a87-b36ca39af526-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:28.656959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.656958 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcz87\" (UniqueName: \"kubernetes.io/projected/22514316-a5c7-439e-8a87-b36ca39af526-kube-api-access-dcz87\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:28.774340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:28.774305 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:29.087009 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:29.086974 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d55bb0_c5ba_45f1_9d18_a123f22a22c8.slice/crio-259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1 WatchSource:0}: Error finding container 259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1: Status 404 returned error can't find the container with id 259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1 Oct 30 21:56:29.087665 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.087601 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj"] Oct 30 21:56:29.337776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.337691 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" Oct 30 21:56:29.337776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.337722 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw" event={"ID":"22514316-a5c7-439e-8a87-b36ca39af526","Type":"ContainerDied","Data":"b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501"} Oct 30 21:56:29.338223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.337782 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b16bd4f0ba37253886d38da0ff997a13340688b31be223a7ca829666dea0f501" Oct 30 21:56:29.339542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.339517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" event={"ID":"42d55bb0-c5ba-45f1-9d18-a123f22a22c8","Type":"ContainerStarted","Data":"259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1"} Oct 30 21:56:29.566139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.566113 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:29.676190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.676158 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\") pod \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " Oct 30 21:56:29.676498 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.676471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\") pod \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " Oct 30 21:56:29.676582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.676568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\" (UniqueName: \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\") pod \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\" (UID: \"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf\") " Oct 30 21:56:29.676919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.676859 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca" pod "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" (UID: "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:29.677542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.677510 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.678621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.678598 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca" pod "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" (UID: "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:29.679208 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.679184 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d" pod "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" (UID: "1b5f66d3-163b-4baf-93bf-73cf39fcc7bf"). InnerVolumeSpecName "kube-api-access-s7h8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:29.779458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.779418 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7h8d\" (UniqueName: \"kubernetes.io/projected/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-kube-api-access-s7h8d\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.779458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:29.779451 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:30.350820 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:30.350791 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" Oct 30 21:56:30.351255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:30.350828 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5" event={"ID":"1b5f66d3-163b-4baf-93bf-73cf39fcc7bf","Type":"ContainerDied","Data":"788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8"} Oct 30 21:56:30.351255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:30.350881 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788653c8d876bea6229bfa258cb76aaf33595f57a053d7dac6acc9532af597a8" Oct 30 21:56:30.352924 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:30.352898 5036 generic.go:356] "Generic (PLEG): container finished" podID="42d55bb0-c5ba-45f1-9d18-a123f22a22c8" containerID="fc401cb428ec5b285a53f7104f2a20e8270a5e335c0ab2c6dba724db24fee09c" exitCode=0 Oct 30 21:56:30.353087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:30.353029 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" event={"ID":"42d55bb0-c5ba-45f1-9d18-a123f22a22c8","Type":"ContainerDied","Data":"fc401cb428ec5b285a53f7104f2a20e8270a5e335c0ab2c6dba724db24fee09c"} Oct 30 21:56:31.052463 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.052428 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf"] Oct 30 21:56:31.058542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.058516 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48902a9d-jl9cf"] Oct 30 21:56:31.208845 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.208808 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" containerID="cri-o://58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e" gracePeriod=25 Oct 30 21:56:31.239682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.239640 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" containerID="cri-o://743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385" gracePeriod=25 Oct 30 21:56:31.602419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.602386 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:31.724822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.724787 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\") pod \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " Oct 30 21:56:31.724994 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.724861 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\") pod \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " Oct 30 21:56:31.725066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.725052 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\" (UniqueName: \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\") pod \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\" (UID: \"42d55bb0-c5ba-45f1-9d18-a123f22a22c8\") " Oct 30 21:56:31.725148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.725126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca" pod "42d55bb0-c5ba-45f1-9d18-a123f22a22c8" (UID: "42d55bb0-c5ba-45f1-9d18-a123f22a22c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:31.725670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.725647 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:31.727185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.727159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca" pod "42d55bb0-c5ba-45f1-9d18-a123f22a22c8" (UID: "42d55bb0-c5ba-45f1-9d18-a123f22a22c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:31.727315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.727258 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw" pod "42d55bb0-c5ba-45f1-9d18-a123f22a22c8" (UID: "42d55bb0-c5ba-45f1-9d18-a123f22a22c8"). InnerVolumeSpecName "kube-api-access-m49kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:31.827325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.827295 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49kw\" (UniqueName: \"kubernetes.io/projected/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-kube-api-access-m49kw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:31.827325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:31.827323 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d55bb0-c5ba-45f1-9d18-a123f22a22c8-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:32.188673 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:32.188643 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c9727f-099a-46ac-9814-2e23e7e53bc2" path="/var/lib/kubelet/pods/27c9727f-099a-46ac-9814-2e23e7e53bc2/volumes" Oct 30 21:56:32.375619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:32.375579 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" event={"ID":"42d55bb0-c5ba-45f1-9d18-a123f22a22c8","Type":"ContainerDied","Data":"259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1"} Oct 30 21:56:32.375783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:32.375644 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259a268cf5d70b2d158af100e7913e9700ef678f9cc6123ce64402d6948cebc1" Oct 30 21:56:32.375783 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:32.375597 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj" Oct 30 21:56:34.135790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:34.135756 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:34.136221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:34.135896 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:35.986337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:35.986300 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:56:36.475291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:36.458216 5036 scope.go:122] "RemoveContainer" containerID="b11d98e7de498de7eecc97ef7eeec1382225c42b7a81b3a143bd44a50b133234" Oct 30 21:56:36.496830 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:36.496799 5036 scope.go:122] "RemoveContainer" containerID="6cfcf88db80db529b37357d51781e9030df8483be768894c41c51a40524b031c" Oct 30 21:56:36.510009 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:36.509983 5036 scope.go:122] "RemoveContainer" containerID="dd348b5921a48bf7aaf3ab137b0a36d88bb16507837fe28fd7076e4c8ace7a72" Oct 30 21:56:40.323893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:40.323859 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9"] Oct 30 21:56:40.329604 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:40.329572 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-rlvw9"] Oct 30 21:56:40.987288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:40.987220 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" podUID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" containerName="payload-processing" containerID="cri-o://212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913" gracePeriod=25 Oct 30 21:56:41.303236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.303205 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:56:41.304786 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.304756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\" (UniqueName: \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\") pod \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " Oct 30 21:56:41.304906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.304821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\") pod \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\" (UID: \"c42024c8-aed2-42ce-bf64-2c2c5759e0b6\") " Oct 30 21:56:41.305228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.305202 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume" pod "c42024c8-aed2-42ce-bf64-2c2c5759e0b6" (UID: "c42024c8-aed2-42ce-bf64-2c2c5759e0b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:41.305448 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.305430 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.307116 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.307088 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg" pod "c42024c8-aed2-42ce-bf64-2c2c5759e0b6" (UID: "c42024c8-aed2-42ce-bf64-2c2c5759e0b6"). InnerVolumeSpecName "kube-api-access-vr7xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:41.406777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.406736 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr7xg\" (UniqueName: \"kubernetes.io/projected/c42024c8-aed2-42ce-bf64-2c2c5759e0b6-kube-api-access-vr7xg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.508518 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.508482 5036 generic.go:356] "Generic (PLEG): container finished" podID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" containerID="212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913" exitCode=0 Oct 30 21:56:41.508715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.508526 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" Oct 30 21:56:41.508715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.508526 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" event={"ID":"c42024c8-aed2-42ce-bf64-2c2c5759e0b6","Type":"ContainerDied","Data":"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913"} Oct 30 21:56:41.508715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.508647 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9" event={"ID":"c42024c8-aed2-42ce-bf64-2c2c5759e0b6","Type":"ContainerDied","Data":"4790c865353335843a49ff44de2ba87a99edbf4786ff2bc6d92926758ac54a2f"} Oct 30 21:56:41.508715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.508682 5036 scope.go:122] "RemoveContainer" containerID="212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913" Oct 30 21:56:41.567206 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.567078 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:56:41.577684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.577570 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c9c95-66f4c66f7c-pg5h9"] Oct 30 21:56:41.577854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.577780 5036 scope.go:122] "RemoveContainer" containerID="212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913" Oct 30 21:56:41.578343 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:56:41.578312 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913\": container with ID starting with 212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913 not found: ID does not exist" containerID="212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913" Oct 30 21:56:41.578542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:41.578354 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913"} err="failed to get container status \"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913\": rpc error: code = NotFound desc = could not find container \"212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913\": container with ID starting with 212cfadc3684e864ac54b2bd2d6dc7c50b94f69fe1bc6e9b0f87ed142c251913 not found: ID does not exist" Oct 30 21:56:42.190708 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.190676 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1489954b-e848-4e34-ad38-c896be27ce36" path="/var/lib/kubelet/pods/1489954b-e848-4e34-ad38-c896be27ce36/volumes" Oct 30 21:56:42.191080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.191066 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" path="/var/lib/kubelet/pods/c42024c8-aed2-42ce-bf64-2c2c5759e0b6/volumes" Oct 30 21:56:42.697634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.697598 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:56:42.698610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698588 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" containerName="revoke-keys" Oct 30 21:56:42.698610 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698608 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" containerName="revoke-keys" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698626 5036 cpu_manager.go:414] "containerMap: removing container" podUID="42d55bb0-c5ba-45f1-9d18-a123f22a22c8" containerName="revoke-keys" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698634 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d55bb0-c5ba-45f1-9d18-a123f22a22c8" containerName="revoke-keys" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698708 5036 cpu_manager.go:414] "containerMap: removing container" podUID="22514316-a5c7-439e-8a87-b36ca39af526" containerName="revoke-keys" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698719 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22514316-a5c7-439e-8a87-b36ca39af526" containerName="revoke-keys" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698749 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" containerName="payload-processing" Oct 30 21:56:42.698878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698757 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" containerName="payload-processing" Oct 30 21:56:42.699307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698916 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" containerName="revoke-keys" Oct 30 21:56:42.699307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698928 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d55bb0-c5ba-45f1-9d18-a123f22a22c8" containerName="revoke-keys" Oct 30 21:56:42.699307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698943 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="22514316-a5c7-439e-8a87-b36ca39af526" containerName="revoke-keys" Oct 30 21:56:42.699307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.698956 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42024c8-aed2-42ce-bf64-2c2c5759e0b6" containerName="payload-processing" Oct 30 21:56:42.704920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.704898 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.712144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.712122 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-03f0a0-openshift-default-dockercfg-dbt2t\"" Oct 30 21:56:42.724226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.724202 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:56:42.724837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.724817 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.724896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.724857 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.724896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.724877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.724974 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.724937 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpz7v\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.725096 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.725075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.725171 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.725153 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.725234 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.725205 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.725484 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.725466 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.725585 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.725510 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.826946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.826909 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.826966 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827015 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827059 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpz7v\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827209 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827255 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827454 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827341 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827754 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.827899 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.827849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.828038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.828019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.829580 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.829556 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.830117 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.830098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.843736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.843715 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:42.843873 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:42.843851 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpz7v\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\") pod \"e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:43.024172 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.024088 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:43.356159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.356130 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:56:43.357892 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:43.357860 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b880d4_a670_414e_a0fe_f09c66e92f3d.slice/crio-c6699fd37ac5384654dbedc433236eeef39f6cd64912c7ac0915ea76463fe4ef WatchSource:0}: Error finding container c6699fd37ac5384654dbedc433236eeef39f6cd64912c7ac0915ea76463fe4ef: Status 404 returned error can't find the container with id c6699fd37ac5384654dbedc433236eeef39f6cd64912c7ac0915ea76463fe4ef Oct 30 21:56:43.360835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.360796 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:43.360930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.360904 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:43.360966 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.360949 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:43.543193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.543154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerStarted","Data":"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7"} Oct 30 21:56:43.543431 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.543220 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerStarted","Data":"c6699fd37ac5384654dbedc433236eeef39f6cd64912c7ac0915ea76463fe4ef"} Oct 30 21:56:43.577485 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:43.577437 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podStartSLOduration=1.577422603 podStartE2EDuration="1.577422603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:43.570730858 +0000 UTC m=+1027.520685818" watchObservedRunningTime="2026-10-30 21:56:43.577422603 +0000 UTC m=+1027.527377548" Oct 30 21:56:44.025093 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.025048 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:44.030898 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.030871 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx"] Oct 30 21:56:44.032013 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.031989 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:44.038561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.038539 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4b17fb-tvpsx"] Oct 30 21:56:44.135978 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.135824 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:44.135978 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.135907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:44.190234 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.190201 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08153fd3-a1bb-43d5-9643-ba875cf0d511" path="/var/lib/kubelet/pods/08153fd3-a1bb-43d5-9643-ba875cf0d511/volumes" Oct 30 21:56:44.552538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.552509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:44.553532 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:44.553516 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:56:45.622166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.622135 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:56:45.627598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.627572 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.641301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.641245 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c9c95-openshift-default-dockercfg-whdd6\"" Oct 30 21:56:45.671323 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.660870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:56:45.690827 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.690649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljknl\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.692153 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.691842 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.692852 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.692416 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.693393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.693374 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.694539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.694516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.694966 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.694950 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.695139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.695122 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.695569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.695546 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.739312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.731513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.832952 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.832879 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljknl\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.832952 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.832946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.832971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833019 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833068 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833182 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833210 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833436 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833447 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833640 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833640 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833563 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833755 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833694 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833831 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.833854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.833845 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.835620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.835600 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.836119 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.836101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.846336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.846309 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.855254 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.855233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljknl\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:45.943380 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:45.943345 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:46.262534 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:46.262503 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2909d6b_c8a6_43f7_83f9_9c1913145a9c.slice/crio-9ff59e316b0f1b4deeb15f4610a92192d2349c831289a91445966d34e93528ad WatchSource:0}: Error finding container 9ff59e316b0f1b4deeb15f4610a92192d2349c831289a91445966d34e93528ad: Status 404 returned error can't find the container with id 9ff59e316b0f1b4deeb15f4610a92192d2349c831289a91445966d34e93528ad Oct 30 21:56:46.266327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.266240 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:46.266419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.266330 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:56:46.266419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.266374 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:46.266495 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.266435 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:46.579907 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.579817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" event={"ID":"b2909d6b-c8a6-43f7-83f9-9c1913145a9c","Type":"ContainerStarted","Data":"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c"} Oct 30 21:56:46.579907 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.579872 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" event={"ID":"b2909d6b-c8a6-43f7-83f9-9c1913145a9c","Type":"ContainerStarted","Data":"9ff59e316b0f1b4deeb15f4610a92192d2349c831289a91445966d34e93528ad"} Oct 30 21:56:46.622448 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.622394 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podStartSLOduration=1.622380838 podStartE2EDuration="1.622380838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:46.617277757 +0000 UTC m=+1030.567232696" watchObservedRunningTime="2026-10-30 21:56:46.622380838 +0000 UTC m=+1030.572335784" Oct 30 21:56:46.943684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:46.943537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:47.057289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:47.055899 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2"] Oct 30 21:56:47.062284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:47.062117 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a23a8ee-95qv2"] Oct 30 21:56:47.957287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:47.956413 5036 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:56:47.957287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:47.956479 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:56:48.105220 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:56:48.105186 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:56:48.191601 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:48.191564 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61634f8d-5849-4ef1-a362-f1d093ba1154" path="/var/lib/kubelet/pods/61634f8d-5849-4ef1-a362-f1d093ba1154/volumes" Oct 30 21:56:48.944189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:48.944154 5036 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:56:48.944503 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:48.944476 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:56:49.791908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.791879 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn"] Oct 30 21:56:49.798461 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.798438 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:49.823299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.810216 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:49.823299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.810307 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:56:49.823299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.810374 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:49.823299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.810417 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:49.865638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.865608 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn"] Oct 30 21:56:49.900041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.900012 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:49.900232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.900146 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smttb\" (UniqueName: \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:49.900343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.900231 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:49.951290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.947811 5036 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:56:49.951290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:49.947895 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:56:50.010293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.009380 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.010293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.009542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smttb\" (UniqueName: \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.010293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.009612 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.030162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.010955 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.048609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.048073 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.129843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.127312 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg"] Oct 30 21:56:50.144287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.139401 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smttb\" (UniqueName: \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\") pod \"maas-api-revoke-sub-452a6aee-7zdzn\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.175296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.174528 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:50.277303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.276743 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d151f-t5nvg"] Oct 30 21:56:50.282667 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.282639 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5"] Oct 30 21:56:50.292620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.292550 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c9c95-trpp5"] Oct 30 21:56:50.609925 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.609899 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn"] Oct 30 21:56:50.854315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.853545 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" event={"ID":"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f","Type":"ContainerStarted","Data":"69db21bc59ccb53fd843619dce1537b90912bc887edbfab97262d596797ca335"} Oct 30 21:56:50.945145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.944657 5036 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:56:50.945145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:50.944718 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:56:51.665681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.665652 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:56:51.678358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.678335 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:56:51.809055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\") pod \"f3e94876-1910-4bec-85ce-33d49517cad0\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " Oct 30 21:56:51.809301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809282 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\" (UniqueName: \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\") pod \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " Oct 30 21:56:51.809377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809363 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\") pod \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\" (UID: \"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06\") " Oct 30 21:56:51.809415 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume" pod "f3e94876-1910-4bec-85ce-33d49517cad0" (UID: "f3e94876-1910-4bec-85ce-33d49517cad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:51.809450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809415 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\" (UniqueName: \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\") pod \"f3e94876-1910-4bec-85ce-33d49517cad0\" (UID: \"f3e94876-1910-4bec-85ce-33d49517cad0\") " Oct 30 21:56:51.810065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.809769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume" pod "a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" (UID: "a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:51.810065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.810031 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e94876-1910-4bec-85ce-33d49517cad0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:51.810065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.810045 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:51.823290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.814850 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65" pod "a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" (UID: "a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06"). InnerVolumeSpecName "kube-api-access-s7l65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:51.823290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.815217 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w" pod "f3e94876-1910-4bec-85ce-33d49517cad0" (UID: "f3e94876-1910-4bec-85ce-33d49517cad0"). InnerVolumeSpecName "kube-api-access-l4j9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:51.865329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.865296 5036 generic.go:356] "Generic (PLEG): container finished" podID="f3e94876-1910-4bec-85ce-33d49517cad0" containerID="743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385" exitCode=0 Oct 30 21:56:51.865704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.865377 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" event={"ID":"f3e94876-1910-4bec-85ce-33d49517cad0","Type":"ContainerDied","Data":"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385"} Oct 30 21:56:51.865704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.865485 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" event={"ID":"f3e94876-1910-4bec-85ce-33d49517cad0","Type":"ContainerDied","Data":"f374863ce6b1f99ee812a5f0b4489f11065f9ef937f9dc18835317722ba0fb4b"} Oct 30 21:56:51.865704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.865489 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj" Oct 30 21:56:51.865704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.865509 5036 scope.go:122] "RemoveContainer" containerID="743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385" Oct 30 21:56:51.867428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.867312 5036 generic.go:356] "Generic (PLEG): container finished" podID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerID="58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e" exitCode=0 Oct 30 21:56:51.867428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.867350 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" Oct 30 21:56:51.867428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.867395 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" event={"ID":"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06","Type":"ContainerDied","Data":"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e"} Oct 30 21:56:51.867573 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.867436 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp" event={"ID":"a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06","Type":"ContainerDied","Data":"04b0f52c597a0bf7e4ce5d22bce2d87ab4cc1b224a4974e0123f49867d813c5e"} Oct 30 21:56:51.869893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.869626 5036 generic.go:356] "Generic (PLEG): container finished" podID="2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" containerID="fcb948ba0c3334f754575ce9e74be8e6131579519dc78587526e758dae91a556" exitCode=0 Oct 30 21:56:51.869893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.869806 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" event={"ID":"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f","Type":"ContainerDied","Data":"fcb948ba0c3334f754575ce9e74be8e6131579519dc78587526e758dae91a556"} Oct 30 21:56:51.893567 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.893547 5036 scope.go:122] "RemoveContainer" containerID="743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385" Oct 30 21:56:51.893871 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:56:51.893852 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385\": container with ID starting with 743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385 not found: ID does not exist" containerID="743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385" Oct 30 21:56:51.893923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.893880 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385"} err="failed to get container status \"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385\": rpc error: code = NotFound desc = could not find container \"743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385\": container with ID starting with 743a3d708bb75250a46a139c3c0fd79b460bb3e81c6371093af44b706ccdc385 not found: ID does not exist" Oct 30 21:56:51.893923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.893898 5036 scope.go:122] "RemoveContainer" containerID="58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e" Oct 30 21:56:51.904349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.904326 5036 scope.go:122] "RemoveContainer" containerID="58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e" Oct 30 21:56:51.904668 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:56:51.904648 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e\": container with ID starting with 58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e not found: ID does not exist" containerID="58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e" Oct 30 21:56:51.904746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.904675 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e"} err="failed to get container status \"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e\": rpc error: code = NotFound desc = could not find container \"58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e\": container with ID starting with 58bc53b9339586bd7ceb7037fd87d2d10d5c1820df2ec2a3ab484895e1084a1e not found: ID does not exist" Oct 30 21:56:51.918297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.911388 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7l65\" (UniqueName: \"kubernetes.io/projected/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06-kube-api-access-s7l65\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:51.918297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.911419 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4j9w\" (UniqueName: \"kubernetes.io/projected/f3e94876-1910-4bec-85ce-33d49517cad0-kube-api-access-l4j9w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:51.949122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.948432 5036 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:56:51.949122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.948486 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:56:51.953349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.953324 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:56:51.958550 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.958528 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-8hqjj"] Oct 30 21:56:51.969184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.969159 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:56:51.975111 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:51.975090 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-gpjdp"] Oct 30 21:56:52.190236 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.190201 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5f66d3-163b-4baf-93bf-73cf39fcc7bf" path="/var/lib/kubelet/pods/1b5f66d3-163b-4baf-93bf-73cf39fcc7bf/volumes" Oct 30 21:56:52.190797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.190774 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" path="/var/lib/kubelet/pods/a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06/volumes" Oct 30 21:56:52.191286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.191248 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd40a60b-57a1-4049-9b08-532c103cc575" path="/var/lib/kubelet/pods/cd40a60b-57a1-4049-9b08-532c103cc575/volumes" Oct 30 21:56:52.192033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.192010 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" path="/var/lib/kubelet/pods/f3e94876-1910-4bec-85ce-33d49517cad0/volumes" Oct 30 21:56:52.950335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.950311 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:52.951961 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.951812 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:52.956851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:52.956802 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:56:53.070045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.070022 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9"] Oct 30 21:56:53.078545 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.078513 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ace82-77cr9"] Oct 30 21:56:53.107034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.106988 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:53.244179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.244091 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:56:53.245317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245278 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" containerName="revoke-keys" Oct 30 21:56:53.245317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245303 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" containerName="revoke-keys" Oct 30 21:56:53.245447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245340 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" Oct 30 21:56:53.245447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245348 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" Oct 30 21:56:53.245447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245384 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" Oct 30 21:56:53.245447 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245392 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" Oct 30 21:56:53.245649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245570 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f5e54d-2389-4c89-a3bf-2d49cf9a9c06" containerName="payload-processing" Oct 30 21:56:53.245649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245592 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" containerName="revoke-keys" Oct 30 21:56:53.245649 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.245604 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e94876-1910-4bec-85ce-33d49517cad0" containerName="payload-pre-processing" Oct 30 21:56:53.250629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.250601 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.270140 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.270116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\" (UniqueName: \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\") pod \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " Oct 30 21:56:53.270321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.270157 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\") pod \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " Oct 30 21:56:53.270321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.270198 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\") pod \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\" (UID: \"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f\") " Oct 30 21:56:53.271637 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.271600 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca" pod "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" (UID: "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:53.279923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.279876 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca" pod "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" (UID: "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:53.280145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.280058 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:56:53.282772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.282740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb" pod "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" (UID: "2f85b7f8-366d-42fa-9c7a-aad4b96cb58f"). InnerVolumeSpecName "kube-api-access-smttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:53.325524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.325477 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-689a2563-dockercfg-fwpqw\"" Oct 30 21:56:53.342511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.342485 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-689a2563\"" Oct 30 21:56:53.372057 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.372025 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvgs\" (UniqueName: \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.372207 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.372186 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.372535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.372518 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smttb\" (UniqueName: \"kubernetes.io/projected/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-kube-api-access-smttb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.372578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.372542 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.372578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.372556 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.473964 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.473936 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvgs\" (UniqueName: \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.474134 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.474016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.474798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.474777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.490869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.490842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvgs\" (UniqueName: \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\") pod \"payload-processing-e2e-mt-689a2563-5df8694775-8rk8l\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.591337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.585569 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:53.892661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.892635 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" Oct 30 21:56:53.892828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.892658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn" event={"ID":"2f85b7f8-366d-42fa-9c7a-aad4b96cb58f","Type":"ContainerDied","Data":"69db21bc59ccb53fd843619dce1537b90912bc887edbfab97262d596797ca335"} Oct 30 21:56:53.892828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.892715 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69db21bc59ccb53fd843619dce1537b90912bc887edbfab97262d596797ca335" Oct 30 21:56:53.991740 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:53.991712 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:56:53.992160 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:53.991815 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b023774_5308_47c2_9a60_f8bd67339f85.slice/crio-77a8bb17ee99b10a18d817a577b8fa552130506364450eae0329ddaf60b74088 WatchSource:0}: Error finding container 77a8bb17ee99b10a18d817a577b8fa552130506364450eae0329ddaf60b74088: Status 404 returned error can't find the container with id 77a8bb17ee99b10a18d817a577b8fa552130506364450eae0329ddaf60b74088 Oct 30 21:56:54.190327 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:54.190199 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad559795-c831-4c7a-bdb0-2b7d63b9d881" path="/var/lib/kubelet/pods/ad559795-c831-4c7a-bdb0-2b7d63b9d881/volumes" Oct 30 21:56:54.931562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:54.931522 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" event={"ID":"1b023774-5308-47c2-9a60-f8bd67339f85","Type":"ContainerStarted","Data":"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49"} Oct 30 21:56:54.931759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:54.931594 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:56:54.931759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:54.931616 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" event={"ID":"1b023774-5308-47c2-9a60-f8bd67339f85","Type":"ContainerStarted","Data":"77a8bb17ee99b10a18d817a577b8fa552130506364450eae0329ddaf60b74088"} Oct 30 21:56:54.958148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:54.958084 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" podStartSLOduration=1.958063924 podStartE2EDuration="1.958063924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:54.95208134 +0000 UTC m=+1038.902036287" watchObservedRunningTime="2026-10-30 21:56:54.958063924 +0000 UTC m=+1038.908018871" Oct 30 21:56:58.382291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.371041 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:58.467292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.467028 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:58.671336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.666330 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:56:58.671336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.666587 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" containerID="cri-o://68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c" gracePeriod=30 Oct 30 21:56:58.673041 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.672979 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb"] Oct 30 21:56:58.689448 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.689426 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb"] Oct 30 21:56:58.689703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.689684 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.693530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.693510 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:58.700312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.693766 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:58.708471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.708438 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:58.708603 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.708585 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:56:58.767735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.767192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4z2p\" (UniqueName: \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.767735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.767420 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.767735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.767587 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.876183 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.876145 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.877580 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.877554 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.877798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.877777 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4z2p\" (UniqueName: \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.878404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.876827 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:56:58.878574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.878547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:58.900556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.900521 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.904298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.901544 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.924331 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.910343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4z2p\" (UniqueName: \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\") pod \"maas-api-revoke-keys-e2e-api-b-79724b-6f8lb\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:58.924331 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:58.918688 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:56:59.653080 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:56:59.653042 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a1dbe1_2a40_4cc0_801e_ee1f5809fa40.slice/crio-2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b WatchSource:0}: Error finding container 2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b: Status 404 returned error can't find the container with id 2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b Oct 30 21:56:59.654189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:56:59.654164 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb"] Oct 30 21:57:00.613363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.612863 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" event={"ID":"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40","Type":"ContainerStarted","Data":"a4450e0839b4634c53cee239726dfe61a4958f5e3c948178a2bfecc5dc80388c"} Oct 30 21:57:00.613363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.612930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" event={"ID":"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40","Type":"ContainerStarted","Data":"2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b"} Oct 30 21:57:00.636882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.636821 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" podStartSLOduration=2.636802507 podStartE2EDuration="2.636802507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:00.630633725 +0000 UTC m=+1044.580588673" watchObservedRunningTime="2026-10-30 21:57:00.636802507 +0000 UTC m=+1044.586757454" Oct 30 21:57:00.850312 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.846481 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:57:00.853177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.853151 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.857766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.857571 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ee0c56-openshift-default-dockercfg-z5xl2\"" Oct 30 21:57:00.863756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.863686 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:57:00.871450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871426 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871463 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871600 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871659 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871723 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871761 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.871916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.871831 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972659 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972706 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972743 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.972872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972828 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.972888 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973020 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973077 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973045 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973243 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973097 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973243 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973243 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973514 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973494 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.973752 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.973733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.976091 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.976057 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:00.976233 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:00.976190 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:01.014051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.014026 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:01.020813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.020785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\") pod \"e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:01.122159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.121544 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs"] Oct 30 21:57:01.175305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.169223 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e627f884-q95fs"] Oct 30 21:57:01.185315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.183568 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:01.583549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.583520 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:57:01.595416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.595374 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:01.595527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.595445 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:01.595527 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.595475 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:01.627757 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.627725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerStarted","Data":"9a31f08d68fcbdea39a66dc5538557907c227fe50beaf17cdd1ac64d2029ba67"} Oct 30 21:57:01.630315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.630146 5036 generic.go:356] "Generic (PLEG): container finished" podID="92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" containerID="a4450e0839b4634c53cee239726dfe61a4958f5e3c948178a2bfecc5dc80388c" exitCode=0 Oct 30 21:57:01.630315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:01.630212 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" event={"ID":"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40","Type":"ContainerDied","Data":"a4450e0839b4634c53cee239726dfe61a4958f5e3c948178a2bfecc5dc80388c"} Oct 30 21:57:02.190180 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:02.190137 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679e9c90-c1a7-4e6b-a149-3e5323c01f60" path="/var/lib/kubelet/pods/679e9c90-c1a7-4e6b-a149-3e5323c01f60/volumes" Oct 30 21:57:02.714325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:02.705308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerStarted","Data":"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284"} Oct 30 21:57:02.735189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:02.735117 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podStartSLOduration=2.73509707 podStartE2EDuration="2.73509707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:02.732382017 +0000 UTC m=+1046.682336965" watchObservedRunningTime="2026-10-30 21:57:02.73509707 +0000 UTC m=+1046.685052016" Oct 30 21:57:03.006173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.006145 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:57:03.030938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.030910 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\") pod \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " Oct 30 21:57:03.031115 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.031011 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\") pod \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " Oct 30 21:57:03.031115 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.031053 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\" (UniqueName: \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\") pod \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\" (UID: \"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40\") " Oct 30 21:57:03.031625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.031599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca" pod "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" (UID: "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:03.048927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.048894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p" pod "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" (UID: "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40"). InnerVolumeSpecName "kube-api-access-x4z2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:03.049069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.048978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca" pod "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" (UID: "92a1dbe1-2a40-4cc0-801e-ee1f5809fa40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:03.133381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.133345 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.133381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.133384 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4z2p\" (UniqueName: \"kubernetes.io/projected/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-kube-api-access-x4z2p\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.133581 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.133401 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.185784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.185758 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:03.191165 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.191140 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:03.721107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.721077 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" Oct 30 21:57:03.721107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.721092 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb" event={"ID":"92a1dbe1-2a40-4cc0-801e-ee1f5809fa40","Type":"ContainerDied","Data":"2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b"} Oct 30 21:57:03.721354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.721140 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbfd6d71a0a6f4062d03a9b98224d2fc92c48a3a0213a042fa92d08c7ee012b" Oct 30 21:57:03.721444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.721425 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:03.722768 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.722747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:03.970296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:03.969879 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:57:04.048185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048146 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.048861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048226 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.048861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048283 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.048861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.048861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048403 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.048861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.048501 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.049147 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.049040 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.049240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.049218 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.049711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.049453 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.051998 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.051971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "kube-api-access-ljknl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.052085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.052031 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.052396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.052367 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.053372 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.053351 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.053449 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.053414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.053522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.053507 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\") pod \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\" (UID: \"b2909d6b-c8a6-43f7-83f9-9c1913145a9c\") " Oct 30 21:57:04.053671 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.053647 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.053762 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.053715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:04.054138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054124 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054145 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054162 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054175 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054190 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054204 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljknl\" (UniqueName: \"kubernetes.io/projected/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-kube-api-access-ljknl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054218 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.054306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.054231 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.055569 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.055548 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo" pod "b2909d6b-c8a6-43f7-83f9-9c1913145a9c" (UID: "b2909d6b-c8a6-43f7-83f9-9c1913145a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:04.155923 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.155885 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2909d6b-c8a6-43f7-83f9-9c1913145a9c-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.732064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.732030 5036 generic.go:356] "Generic (PLEG): container finished" podID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerID="68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c" exitCode=0 Oct 30 21:57:04.732482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.732077 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" Oct 30 21:57:04.732482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.732110 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" event={"ID":"b2909d6b-c8a6-43f7-83f9-9c1913145a9c","Type":"ContainerDied","Data":"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c"} Oct 30 21:57:04.732482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.732165 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb" event={"ID":"b2909d6b-c8a6-43f7-83f9-9c1913145a9c","Type":"ContainerDied","Data":"9ff59e316b0f1b4deeb15f4610a92192d2349c831289a91445966d34e93528ad"} Oct 30 21:57:04.732482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.732187 5036 scope.go:122] "RemoveContainer" containerID="68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c" Oct 30 21:57:04.748889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.745322 5036 scope.go:122] "RemoveContainer" containerID="68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c" Oct 30 21:57:04.748889 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:04.745845 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c\": container with ID starting with 68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c not found: ID does not exist" containerID="68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c" Oct 30 21:57:04.748889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.745878 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c"} err="failed to get container status \"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c\": rpc error: code = NotFound desc = could not find container \"68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c\": container with ID starting with 68cc3c6e856aa51911e0666a08541692f8b9a76c8da9e0e8a67cdd76b551277c not found: ID does not exist" Oct 30 21:57:04.759390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.759336 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:57:04.766880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:04.766852 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sglkb"] Oct 30 21:57:05.960008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:05.959979 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:57:06.196821 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:06.196789 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" path="/var/lib/kubelet/pods/b2909d6b-c8a6-43f7-83f9-9c1913145a9c/volumes" Oct 30 21:57:08.027382 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:08.027341 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk"] Oct 30 21:57:08.032639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:08.032605 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f88ce9-9b4dk"] Oct 30 21:57:08.189428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:08.189389 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c505b248-de69-468b-83b2-f199f7a8e286" path="/var/lib/kubelet/pods/c505b248-de69-468b-83b2-f199f7a8e286/volumes" Oct 30 21:57:12.357199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.357156 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:57:12.358139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358113 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" Oct 30 21:57:12.358139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358139 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" Oct 30 21:57:12.358393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358167 5036 cpu_manager.go:414] "containerMap: removing container" podUID="92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" containerName="revoke-keys" Oct 30 21:57:12.358393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358174 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" containerName="revoke-keys" Oct 30 21:57:12.358492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358453 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2909d6b-c8a6-43f7-83f9-9c1913145a9c" containerName="istio-proxy" Oct 30 21:57:12.358492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.358473 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" containerName="revoke-keys" Oct 30 21:57:12.364406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.364385 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.379494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.379469 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:57:12.383920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.383903 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.386030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.386008 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-56c0af\"" Oct 30 21:57:12.387245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.387219 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.387803 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.387776 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.387938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.387867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5xx\" (UniqueName: \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.388001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.387949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2ln\" (UniqueName: \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.393611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.393592 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:57:12.396689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.396668 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-56c0af-dockercfg-d2d7d\"" Oct 30 21:57:12.405523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.405495 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:57:12.489192 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.489153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.499130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.491678 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5xx\" (UniqueName: \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.499130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.491767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2ln\" (UniqueName: \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.499130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.491984 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.499130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.491541 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.499130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.493134 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.630339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.629992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5xx\" (UniqueName: \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\") pod \"payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.672346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.654329 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2ln\" (UniqueName: \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\") pod \"payload-processing-e2e-api-b-56c0af-598dc77555-zkp64\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:12.685611 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.684327 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:12.720980 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:12.719607 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:13.187965 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:57:13.187936 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ecc44_9246_4ca0_86b9_6e9abd27f4eb.slice/crio-e9b2c4a3298f2e407cd98dcd3603493d2c17db429d041c77fd68c2a7d9383e72 WatchSource:0}: Error finding container e9b2c4a3298f2e407cd98dcd3603493d2c17db429d041c77fd68c2a7d9383e72: Status 404 returned error can't find the container with id e9b2c4a3298f2e407cd98dcd3603493d2c17db429d041c77fd68c2a7d9383e72 Oct 30 21:57:13.188577 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.188552 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:57:13.194078 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:57:13.194049 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b64500_c261_45d6_81ed_c465dd6914a0.slice/crio-16dd8fae387294ff93dcbf149ef47907d72106dafc0ca87d92352ef5168488f4 WatchSource:0}: Error finding container 16dd8fae387294ff93dcbf149ef47907d72106dafc0ca87d92352ef5168488f4: Status 404 returned error can't find the container with id 16dd8fae387294ff93dcbf149ef47907d72106dafc0ca87d92352ef5168488f4 Oct 30 21:57:13.196250 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.196230 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:57:13.875141 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.875106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" event={"ID":"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb","Type":"ContainerStarted","Data":"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7"} Oct 30 21:57:13.875652 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.875159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" event={"ID":"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb","Type":"ContainerStarted","Data":"e9b2c4a3298f2e407cd98dcd3603493d2c17db429d041c77fd68c2a7d9383e72"} Oct 30 21:57:13.875652 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.875198 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:13.877038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.877018 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" event={"ID":"91b64500-c261-45d6-81ed-c465dd6914a0","Type":"ContainerStarted","Data":"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566"} Oct 30 21:57:13.877121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.877051 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" event={"ID":"91b64500-c261-45d6-81ed-c465dd6914a0","Type":"ContainerStarted","Data":"16dd8fae387294ff93dcbf149ef47907d72106dafc0ca87d92352ef5168488f4"} Oct 30 21:57:13.877121 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.877073 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:13.969833 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.969766 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" podStartSLOduration=1.969739355 podStartE2EDuration="1.969739355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:13.937507104 +0000 UTC m=+1057.887462051" watchObservedRunningTime="2026-10-30 21:57:13.969739355 +0000 UTC m=+1057.919694304" Oct 30 21:57:13.973015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:13.972961 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" podStartSLOduration=1.97293752 podStartE2EDuration="1.97293752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:13.964663651 +0000 UTC m=+1057.914618621" watchObservedRunningTime="2026-10-30 21:57:13.97293752 +0000 UTC m=+1057.922892467" Oct 30 21:57:18.163657 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:18.163582 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:57:18.853296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:18.852168 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:57:18.853296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:18.852232 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:19.780287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:19.775083 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:19.826625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:19.826577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:19.905522 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:19.905474 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:20.119356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.106285 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 21:57:20.140301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.139319 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf"] Oct 30 21:57:20.163286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.155360 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:20.244555 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.244508 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 21:57:20.360287 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.354333 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f6e27-znvdf"] Oct 30 21:57:20.501304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.489372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:20.829366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.828513 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:20.851295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:20.842332 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:57:21.101347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.100692 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 21:57:21.140290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.139597 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 21:57:21.149294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.141421 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:57:21.257451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.248355 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:21.257451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.249103 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:57:21.320292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.303612 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:21.396076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.396048 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 21:57:21.401052 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:21.397533 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 21:57:22.230291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:22.203463 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d658eb4-6d6d-4be3-a388-b11caa796ca9" path="/var/lib/kubelet/pods/2d658eb4-6d6d-4be3-a388-b11caa796ca9/volumes" Oct 30 21:57:24.491645 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.491611 5036 generic.go:356] "Generic (PLEG): container finished" podID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerID="0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b" exitCode=0 Oct 30 21:57:24.492079 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.491686 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerDied","Data":"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b"} Oct 30 21:57:24.492247 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.492231 5036 scope.go:122] "RemoveContainer" containerID="0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b" Oct 30 21:57:24.493972 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.493916 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:24.494059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.494018 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:24.494059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.494052 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:24.606034 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.606006 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb"] Oct 30 21:57:24.611819 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.611607 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-79724b-6f8lb"] Oct 30 21:57:24.941643 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.941415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:57:24.944369 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:24.943333 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:57:25.505827 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:25.505793 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerStarted","Data":"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0"} Oct 30 21:57:26.190475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.190439 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a1dbe1-2a40-4cc0-801e-ee1f5809fa40" path="/var/lib/kubelet/pods/92a1dbe1-2a40-4cc0-801e-ee1f5809fa40/volumes" Oct 30 21:57:26.248832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.246850 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:26.248832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.246989 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:26.248832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.247031 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:26.280343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.280011 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:57:26.285185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.285164 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.351292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.348285 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4bf337-openshift-default-dockercfg-qzhd6\"" Oct 30 21:57:26.374551 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.374508 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.374751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.374583 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.374828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.374762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.374893 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.374871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.374968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.374932 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8rk\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.375597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.375041 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.375597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.375074 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:57:26.375597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.375315 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.375597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.375350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.375597 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.375383 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510577 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510727 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510814 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8rk\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.510913 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.511045 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.511073 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.511109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.512149 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.512488 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.512732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.512954 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.526139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.513190 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.703558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.698585 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8rk\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.754928 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.754071 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.754928 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.754555 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.789469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.789399 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\") pod \"e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:26.936298 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:26.934807 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:27.168044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.163618 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq"] Oct 30 21:57:27.177460 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.177196 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.180466 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.180428 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq"] Oct 30 21:57:27.189309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.183852 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:27.189309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.183977 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:57:27.189309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.184674 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:27.189309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.184813 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:27.250054 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.249384 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:27.250054 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.249437 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:27.273986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.273946 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.274172 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.274009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvj5\" (UniqueName: \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.274172 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.274052 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.356523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.356468 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:27.356660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.356628 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:27.356747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.356725 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:27.357869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.357639 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:57:27.379017 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.378979 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.379144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.379048 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvj5\" (UniqueName: \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.379144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.379086 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.379969 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.379934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.381985 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.381904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.442306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.441976 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvj5\" (UniqueName: \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\") pod \"maas-api-revoke-keys-e2e-mt-689a2563-vtnwq\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.504219 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.503454 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:27.807070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.807027 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerStarted","Data":"432c3d6ee7ac53057ed9d918a7eb60908f4a86c46500b203e074554f572aa93f"} Oct 30 21:57:27.807557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.807107 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerStarted","Data":"416006c5825902163a190198f7495e64d28e14036851f9f838535f08b372aefc"} Oct 30 21:57:27.866470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.864165 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podStartSLOduration=1.864146013 podStartE2EDuration="1.864146013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:27.862020986 +0000 UTC m=+1071.811975955" watchObservedRunningTime="2026-10-30 21:57:27.864146013 +0000 UTC m=+1071.814100961" Oct 30 21:57:27.954948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.954913 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:27.969547 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:27.969474 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq"] Oct 30 21:57:28.284462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.284405 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:28.284720 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.284494 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:28.866248 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.866162 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" event={"ID":"cba2c2a1-f451-474c-9c95-b7a0b8256519","Type":"ContainerStarted","Data":"ac38e53afb942c73817c4b2612fcb01658ac08f6f6f0ce8094ffcd9230b9974a"} Oct 30 21:57:28.866879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.866855 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" event={"ID":"cba2c2a1-f451-474c-9c95-b7a0b8256519","Type":"ContainerStarted","Data":"4d9d76b84d511464658b0b56e20897bee8466a12a7f8e68b1b824928af9174b1"} Oct 30 21:57:28.899499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.899441 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" podStartSLOduration=1.8994279440000001 podStartE2EDuration="1.899427944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:28.894896534 +0000 UTC m=+1072.844851511" watchObservedRunningTime="2026-10-30 21:57:28.899427944 +0000 UTC m=+1072.849382889" Oct 30 21:57:28.945804 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:28.945774 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:29.245677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.245644 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:29.245855 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.245698 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:29.878221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.878184 5036 generic.go:356] "Generic (PLEG): container finished" podID="cba2c2a1-f451-474c-9c95-b7a0b8256519" containerID="ac38e53afb942c73817c4b2612fcb01658ac08f6f6f0ce8094ffcd9230b9974a" exitCode=0 Oct 30 21:57:29.878739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.878298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" event={"ID":"cba2c2a1-f451-474c-9c95-b7a0b8256519","Type":"ContainerDied","Data":"ac38e53afb942c73817c4b2612fcb01658ac08f6f6f0ce8094ffcd9230b9974a"} Oct 30 21:57:29.878810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.878788 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:29.879835 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:29.879816 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:57:30.246071 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:30.246037 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:30.246228 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:30.246111 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:30.594479 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:30.594382 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:57:30.594479 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:30.594436 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:31.246968 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.246874 5036 patch_prober.go:29] interesting pod/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:31.262142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.262097 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 21:57:31.347763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.347731 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:31.458732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.458693 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\" (UniqueName: \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\") pod \"cba2c2a1-f451-474c-9c95-b7a0b8256519\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " Oct 30 21:57:31.458912 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.458834 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\") pod \"cba2c2a1-f451-474c-9c95-b7a0b8256519\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " Oct 30 21:57:31.459108 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.459090 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\") pod \"cba2c2a1-f451-474c-9c95-b7a0b8256519\" (UID: \"cba2c2a1-f451-474c-9c95-b7a0b8256519\") " Oct 30 21:57:31.459797 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.459757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca" pod "cba2c2a1-f451-474c-9c95-b7a0b8256519" (UID: "cba2c2a1-f451-474c-9c95-b7a0b8256519"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:31.464322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.463015 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5" pod "cba2c2a1-f451-474c-9c95-b7a0b8256519" (UID: "cba2c2a1-f451-474c-9c95-b7a0b8256519"). InnerVolumeSpecName "kube-api-access-4zvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:31.464322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.463024 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca" pod "cba2c2a1-f451-474c-9c95-b7a0b8256519" (UID: "cba2c2a1-f451-474c-9c95-b7a0b8256519"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:31.561356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.561325 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:31.561356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.561356 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvj5\" (UniqueName: \"kubernetes.io/projected/cba2c2a1-f451-474c-9c95-b7a0b8256519-kube-api-access-4zvj5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:31.561541 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:31.561370 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba2c2a1-f451-474c-9c95-b7a0b8256519-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:32.046288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.042419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" event={"ID":"cba2c2a1-f451-474c-9c95-b7a0b8256519","Type":"ContainerDied","Data":"4d9d76b84d511464658b0b56e20897bee8466a12a7f8e68b1b824928af9174b1"} Oct 30 21:57:32.046288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.042485 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9d76b84d511464658b0b56e20897bee8466a12a7f8e68b1b824928af9174b1" Oct 30 21:57:32.046288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.042554 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq" Oct 30 21:57:32.089289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.088869 5036 generic.go:356] "Generic (PLEG): container finished" podID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerID="e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7" exitCode=0 Oct 30 21:57:32.099945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.089997 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerDied","Data":"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7"} Oct 30 21:57:32.099945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.090476 5036 scope.go:122] "RemoveContainer" containerID="e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7" Oct 30 21:57:32.118705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.118653 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:32.118984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.118958 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:32.119137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.119118 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:32.262542 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.261256 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:32.266118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.264701 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:32.266592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:32.266530 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:33.002958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.002925 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6"] Oct 30 21:57:33.003784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.003762 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cba2c2a1-f451-474c-9c95-b7a0b8256519" containerName="revoke-keys" Oct 30 21:57:33.003784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.003783 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba2c2a1-f451-474c-9c95-b7a0b8256519" containerName="revoke-keys" Oct 30 21:57:33.003933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.003901 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba2c2a1-f451-474c-9c95-b7a0b8256519" containerName="revoke-keys" Oct 30 21:57:33.007763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.007745 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.034292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.026624 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:33.034292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.027405 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:57:33.043494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.042988 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:33.043494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.043084 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:33.063794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.051383 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6"] Oct 30 21:57:33.101916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.101879 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerStarted","Data":"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3"} Oct 30 21:57:33.154350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.154312 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.154508 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.154474 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.154663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.154645 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97p68\" (UniqueName: \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.256469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.256373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97p68\" (UniqueName: \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.256661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.256566 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.256739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.256695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.257385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.257360 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.259386 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.259367 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.285147 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.285092 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97p68\" (UniqueName: \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\") pod \"maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.373401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.373367 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:33.834346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:33.834315 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6"] Oct 30 21:57:33.835240 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:57:33.835212 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee9d175_616a_445d_b9a6_21fb787ceb14.slice/crio-088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4 WatchSource:0}: Error finding container 088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4: Status 404 returned error can't find the container with id 088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4 Oct 30 21:57:34.031562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:34.026688 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:57:34.031562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:34.026811 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:57:34.031562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:34.026853 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:57:34.155132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:34.155093 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" event={"ID":"8ee9d175-616a-445d-b9a6-21fb787ceb14","Type":"ContainerStarted","Data":"088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4"} Oct 30 21:57:34.789173 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:34.789096 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:57:35.036294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:35.032740 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:57:35.036294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:35.032801 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:57:35.176229 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:35.176195 5036 generic.go:356] "Generic (PLEG): container finished" podID="8ee9d175-616a-445d-b9a6-21fb787ceb14" containerID="bf9addbc0d544a95544f4edd08c490f23261f8bd04491c5dff3328e6c5a9a86c" exitCode=0 Oct 30 21:57:35.176420 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:35.176301 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" event={"ID":"8ee9d175-616a-445d-b9a6-21fb787ceb14","Type":"ContainerDied","Data":"bf9addbc0d544a95544f4edd08c490f23261f8bd04491c5dff3328e6c5a9a86c"} Oct 30 21:57:35.176502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:35.176478 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" containerID="cri-o://e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0" gracePeriod=30 Oct 30 21:57:36.025761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.025724 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:57:36.026416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.026374 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:57:36.471901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.471878 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:36.579064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.578977 5036 scope.go:122] "RemoveContainer" containerID="fc4c40aff4a42754ed9ae7fa408eb9d794c8a9f0bd311a87ea938a107668ebf2" Oct 30 21:57:36.595314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.595248 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\") pod \"8ee9d175-616a-445d-b9a6-21fb787ceb14\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " Oct 30 21:57:36.595458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.595341 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\") pod \"8ee9d175-616a-445d-b9a6-21fb787ceb14\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " Oct 30 21:57:36.595688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.595671 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\" (UniqueName: \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\") pod \"8ee9d175-616a-445d-b9a6-21fb787ceb14\" (UID: \"8ee9d175-616a-445d-b9a6-21fb787ceb14\") " Oct 30 21:57:36.596717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.596688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca" pod "8ee9d175-616a-445d-b9a6-21fb787ceb14" (UID: "8ee9d175-616a-445d-b9a6-21fb787ceb14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:36.598653 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.598631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca" pod "8ee9d175-616a-445d-b9a6-21fb787ceb14" (UID: "8ee9d175-616a-445d-b9a6-21fb787ceb14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:36.598801 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.598782 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68" pod "8ee9d175-616a-445d-b9a6-21fb787ceb14" (UID: "8ee9d175-616a-445d-b9a6-21fb787ceb14"). InnerVolumeSpecName "kube-api-access-97p68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:36.698179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.698150 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97p68\" (UniqueName: \"kubernetes.io/projected/8ee9d175-616a-445d-b9a6-21fb787ceb14-kube-api-access-97p68\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:36.698179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.698180 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:36.698404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:36.698198 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9d175-616a-445d-b9a6-21fb787ceb14-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:37.031291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.030372 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:57:37.031291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.030433 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:57:37.198160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.198130 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" Oct 30 21:57:37.198160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.198142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6" event={"ID":"8ee9d175-616a-445d-b9a6-21fb787ceb14","Type":"ContainerDied","Data":"088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4"} Oct 30 21:57:37.198416 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.198191 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088151449c26b74acc550cc5751d28f8aafa5cfa7d22bd44bf1c586456ff23b4" Oct 30 21:57:37.353131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:37.353038 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:57:38.029847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:38.029813 5036 patch_prober.go:29] interesting pod/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:57:38.030142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:38.030112 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:57:39.030429 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:39.030399 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:57:39.030927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:39.030800 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:57:39.031777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:39.031754 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:57:40.483631 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.483607 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:40.576138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.546414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.594790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594379 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:40.594790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594518 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.594790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.594790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594679 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.595215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.595215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.595215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.594979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.595215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.595010 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.600441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.600398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\") pod \"3d948f42-c502-4b96-ae12-d052d17a1c55\" (UID: \"3d948f42-c502-4b96-ae12-d052d17a1c55\") " Oct 30 21:57:40.603258 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.602033 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.604457 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.604409 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:40.604580 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.604423 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:40.605417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.605354 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:40.605551 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.605527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:40.607101 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.607011 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:40.609829 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.609800 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:40.627349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.627179 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "kube-api-access-nh9b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:40.639778 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.639742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy" pod "3d948f42-c502-4b96-ae12-d052d17a1c55" (UID: "3d948f42-c502-4b96-ae12-d052d17a1c55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:40.703505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703431 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703459 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh9b9\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-kube-api-access-nh9b9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703473 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703484 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703497 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703511 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703523 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d948f42-c502-4b96-ae12-d052d17a1c55-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:40.703770 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:40.703534 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d948f42-c502-4b96-ae12-d052d17a1c55-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:41.281996 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.281948 5036 generic.go:356] "Generic (PLEG): container finished" podID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerID="e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0" exitCode=0 Oct 30 21:57:41.282231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.282037 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" Oct 30 21:57:41.282231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.282077 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerDied","Data":"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0"} Oct 30 21:57:41.282231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.282145 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl" event={"ID":"3d948f42-c502-4b96-ae12-d052d17a1c55","Type":"ContainerDied","Data":"ad5a591a7deb2a151b6468c01e2e239d922040962c3f63d2b77191f2b35e3f84"} Oct 30 21:57:41.282231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.282180 5036 scope.go:122] "RemoveContainer" containerID="e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0" Oct 30 21:57:41.312391 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.312364 5036 scope.go:122] "RemoveContainer" containerID="0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b" Oct 30 21:57:41.336440 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.336405 5036 scope.go:122] "RemoveContainer" containerID="e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0" Oct 30 21:57:41.336859 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:41.336831 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0\": container with ID starting with e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0 not found: ID does not exist" containerID="e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0" Oct 30 21:57:41.336956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.336872 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0"} err="failed to get container status \"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0\": rpc error: code = NotFound desc = could not find container \"e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0\": container with ID starting with e1e9372b5336087d11a1042f16a842c4ac1f3f1139d9406626e129f048463cc0 not found: ID does not exist" Oct 30 21:57:41.336956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.336900 5036 scope.go:122] "RemoveContainer" containerID="0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b" Oct 30 21:57:41.337203 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:41.337184 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b\": container with ID starting with 0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b not found: ID does not exist" containerID="0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b" Oct 30 21:57:41.337310 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.337210 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b"} err="failed to get container status \"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b\": rpc error: code = NotFound desc = could not find container \"0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b\": container with ID starting with 0839b0c208abce451c39f6dfe145e034af6da8f5baa286298c001bdce778c86b not found: ID does not exist" Oct 30 21:57:41.339467 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.339445 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:57:41.348251 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:41.348231 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-79724b-openshift-default-69f7bd5cfc-l6gsl"] Oct 30 21:57:42.189765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.189729 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" path="/var/lib/kubelet/pods/3d948f42-c502-4b96-ae12-d052d17a1c55/volumes" Oct 30 21:57:42.354714 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.354672 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" podUID="1b023774-5308-47c2-9a60-f8bd67339f85" containerName="payload-processing" containerID="cri-o://5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49" gracePeriod=25 Oct 30 21:57:42.664766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.664736 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:57:42.843526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.843074 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\" (UniqueName: \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\") pod \"1b023774-5308-47c2-9a60-f8bd67339f85\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " Oct 30 21:57:42.843526 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.843234 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\") pod \"1b023774-5308-47c2-9a60-f8bd67339f85\" (UID: \"1b023774-5308-47c2-9a60-f8bd67339f85\") " Oct 30 21:57:42.843866 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.843836 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume" pod "1b023774-5308-47c2-9a60-f8bd67339f85" (UID: "1b023774-5308-47c2-9a60-f8bd67339f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:42.844131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.844084 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b023774-5308-47c2-9a60-f8bd67339f85-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.846901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.846874 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs" pod "1b023774-5308-47c2-9a60-f8bd67339f85" (UID: "1b023774-5308-47c2-9a60-f8bd67339f85"). InnerVolumeSpecName "kube-api-access-4lvgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:42.945826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:42.945781 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvgs\" (UniqueName: \"kubernetes.io/projected/1b023774-5308-47c2-9a60-f8bd67339f85-kube-api-access-4lvgs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:43.307513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.307477 5036 generic.go:356] "Generic (PLEG): container finished" podID="1b023774-5308-47c2-9a60-f8bd67339f85" containerID="5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49" exitCode=0 Oct 30 21:57:43.308008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.307529 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" Oct 30 21:57:43.308008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.307799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" event={"ID":"1b023774-5308-47c2-9a60-f8bd67339f85","Type":"ContainerDied","Data":"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49"} Oct 30 21:57:43.308008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.307850 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l" event={"ID":"1b023774-5308-47c2-9a60-f8bd67339f85","Type":"ContainerDied","Data":"77a8bb17ee99b10a18d817a577b8fa552130506364450eae0329ddaf60b74088"} Oct 30 21:57:43.308008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.307878 5036 scope.go:122] "RemoveContainer" containerID="5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49" Oct 30 21:57:43.330199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.330179 5036 scope.go:122] "RemoveContainer" containerID="5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49" Oct 30 21:57:43.330592 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:43.330562 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49\": container with ID starting with 5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49 not found: ID does not exist" containerID="5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49" Oct 30 21:57:43.330704 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.330602 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49"} err="failed to get container status \"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49\": rpc error: code = NotFound desc = could not find container \"5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49\": container with ID starting with 5363c589f077551cddf5e48694c6d72f86fc7f74d2971df0b529bbdb127a8e49 not found: ID does not exist" Oct 30 21:57:43.339422 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.339393 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:57:43.344641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.344616 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-689a2563-5df8694775-8rk8l"] Oct 30 21:57:43.741775 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.741745 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq"] Oct 30 21:57:43.750016 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:43.749979 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689a2563-vtnwq"] Oct 30 21:57:44.197946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:44.197914 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b023774-5308-47c2-9a60-f8bd67339f85" path="/var/lib/kubelet/pods/1b023774-5308-47c2-9a60-f8bd67339f85/volumes" Oct 30 21:57:44.198392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:44.198379 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba2c2a1-f451-474c-9c95-b7a0b8256519" path="/var/lib/kubelet/pods/cba2c2a1-f451-474c-9c95-b7a0b8256519/volumes" Oct 30 21:57:48.101253 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:57:48.101221 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:57:51.389862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.389826 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf"] Oct 30 21:57:51.390539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390516 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.390539 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390533 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390553 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390559 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390586 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1b023774-5308-47c2-9a60-f8bd67339f85" containerName="payload-processing" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390593 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b023774-5308-47c2-9a60-f8bd67339f85" containerName="payload-processing" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390599 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8ee9d175-616a-445d-b9a6-21fb787ceb14" containerName="revoke-keys" Oct 30 21:57:51.390703 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390604 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee9d175-616a-445d-b9a6-21fb787ceb14" containerName="revoke-keys" Oct 30 21:57:51.390882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390715 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b023774-5308-47c2-9a60-f8bd67339f85" containerName="payload-processing" Oct 30 21:57:51.390882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390722 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee9d175-616a-445d-b9a6-21fb787ceb14" containerName="revoke-keys" Oct 30 21:57:51.390882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390734 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.390882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.390742 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d948f42-c502-4b96-ae12-d052d17a1c55" containerName="istio-proxy" Oct 30 21:57:51.394589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.394574 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.405960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.405926 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf"] Oct 30 21:57:51.406490 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.406474 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:51.407069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.407052 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:51.446294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.435623 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:57:51.462343 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.462307 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:51.487595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.487567 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.487809 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.487784 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s8s\" (UniqueName: \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.487945 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.487929 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.589879 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.589845 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s8s\" (UniqueName: \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.590073 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.589955 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.590073 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.589998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.590636 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.590613 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.592618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.592595 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.635080 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.635045 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s8s\" (UniqueName: \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\") pod \"maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:51.732626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:51.732528 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:52.028986 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:57:52.028925 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c2f7be_1198_4333_a595_6805d27034ad.slice/crio-e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9 WatchSource:0}: Error finding container e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9: Status 404 returned error can't find the container with id e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9 Oct 30 21:57:52.031107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:52.031077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf"] Oct 30 21:57:52.424348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:52.424313 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" event={"ID":"d8c2f7be-1198-4333-a595-6805d27034ad","Type":"ContainerStarted","Data":"e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9"} Oct 30 21:57:53.030824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:53.030780 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6"] Oct 30 21:57:53.037295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:53.037252 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-03f0a0-hd8z6"] Oct 30 21:57:53.435402 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:53.435343 5036 generic.go:356] "Generic (PLEG): container finished" podID="d8c2f7be-1198-4333-a595-6805d27034ad" containerID="7dc53e76b436167f66f3ef19f0fde23e6cd225621f3d014bdddee9f5d2b690cc" exitCode=0 Oct 30 21:57:53.435844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:53.435425 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" event={"ID":"d8c2f7be-1198-4333-a595-6805d27034ad","Type":"ContainerDied","Data":"7dc53e76b436167f66f3ef19f0fde23e6cd225621f3d014bdddee9f5d2b690cc"} Oct 30 21:57:54.193453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.193417 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee9d175-616a-445d-b9a6-21fb787ceb14" path="/var/lib/kubelet/pods/8ee9d175-616a-445d-b9a6-21fb787ceb14/volumes" Oct 30 21:57:54.636670 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.636636 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:54.650276 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.650225 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\") pod \"d8c2f7be-1198-4333-a595-6805d27034ad\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " Oct 30 21:57:54.650451 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.650421 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\" (UniqueName: \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\") pod \"d8c2f7be-1198-4333-a595-6805d27034ad\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " Oct 30 21:57:54.650561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.650538 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca" pod "d8c2f7be-1198-4333-a595-6805d27034ad" (UID: "d8c2f7be-1198-4333-a595-6805d27034ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:54.651024 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.651002 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.652737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.652713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s" pod "d8c2f7be-1198-4333-a595-6805d27034ad" (UID: "d8c2f7be-1198-4333-a595-6805d27034ad"). InnerVolumeSpecName "kube-api-access-p9s8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:54.792346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.782999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\") pod \"d8c2f7be-1198-4333-a595-6805d27034ad\" (UID: \"d8c2f7be-1198-4333-a595-6805d27034ad\") " Oct 30 21:57:54.792346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.783666 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s8s\" (UniqueName: \"kubernetes.io/projected/d8c2f7be-1198-4333-a595-6805d27034ad-kube-api-access-p9s8s\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.830404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.830333 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca" pod "d8c2f7be-1198-4333-a595-6805d27034ad" (UID: "d8c2f7be-1198-4333-a595-6805d27034ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:54.893488 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:54.892822 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c2f7be-1198-4333-a595-6805d27034ad-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:55.456516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.456481 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" Oct 30 21:57:55.456516 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.456498 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf" event={"ID":"d8c2f7be-1198-4333-a595-6805d27034ad","Type":"ContainerDied","Data":"e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9"} Oct 30 21:57:55.456735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.456558 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2eea188ff1b9b1e681de6e7a81f1eb0f21cc722e324a041a6ed24357cc14ce9" Oct 30 21:57:55.711934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.711852 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6"] Oct 30 21:57:55.712557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.712539 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d8c2f7be-1198-4333-a595-6805d27034ad" containerName="revoke-keys" Oct 30 21:57:55.712618 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.712559 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c2f7be-1198-4333-a595-6805d27034ad" containerName="revoke-keys" Oct 30 21:57:55.712791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.712765 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c2f7be-1198-4333-a595-6805d27034ad" containerName="revoke-keys" Oct 30 21:57:55.717984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.717956 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.725713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.725684 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:57:55.733249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.731415 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:55.733249 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.731488 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:55.740713 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.740683 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6"] Oct 30 21:57:55.809975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.809937 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.809975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.809984 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdhh\" (UniqueName: \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.810237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.810009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.824739 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.824716 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:55.911340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.911286 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.911513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.911351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdhh\" (UniqueName: \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.911513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.911398 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.912421 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.912396 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.914726 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.914538 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:55.924005 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:55.923976 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdhh\" (UniqueName: \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\") pod \"maas-api-revoke-sub-1f26c672-ssfq6\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:56.041661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:56.041583 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:56.349742 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:56.349715 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6"] Oct 30 21:57:56.351498 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:57:56.351466 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec60992_5bf6_426a_b362_f627124db07c.slice/crio-96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3 WatchSource:0}: Error finding container 96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3: Status 404 returned error can't find the container with id 96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3 Oct 30 21:57:56.469458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:56.469425 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" event={"ID":"1ec60992-5bf6-426a-b362-f627124db07c","Type":"ContainerStarted","Data":"96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3"} Oct 30 21:57:57.479979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.479939 5036 generic.go:356] "Generic (PLEG): container finished" podID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerID="91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284" exitCode=0 Oct 30 21:57:57.480442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.480005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerDied","Data":"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284"} Oct 30 21:57:57.480558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.480537 5036 scope.go:122] "RemoveContainer" containerID="91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284" Oct 30 21:57:57.482285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.482249 5036 generic.go:356] "Generic (PLEG): container finished" podID="1ec60992-5bf6-426a-b362-f627124db07c" containerID="da70cc011745add90315264844bf7634d1a64cee16c23dd1a5d77aeb42c6d20d" exitCode=0 Oct 30 21:57:57.482406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.482295 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:57.482406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.482318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" event={"ID":"1ec60992-5bf6-426a-b362-f627124db07c","Type":"ContainerDied","Data":"da70cc011745add90315264844bf7634d1a64cee16c23dd1a5d77aeb42c6d20d"} Oct 30 21:57:57.482406 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.482369 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:57.482538 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:57.482410 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:58.510126 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.510088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerStarted","Data":"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1"} Oct 30 21:57:58.789746 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.789720 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:58.870505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.870474 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\") pod \"1ec60992-5bf6-426a-b362-f627124db07c\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " Oct 30 21:57:58.870705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.870546 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\") pod \"1ec60992-5bf6-426a-b362-f627124db07c\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " Oct 30 21:57:58.870705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.870590 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\" (UniqueName: \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\") pod \"1ec60992-5bf6-426a-b362-f627124db07c\" (UID: \"1ec60992-5bf6-426a-b362-f627124db07c\") " Oct 30 21:57:58.870872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.870846 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca" pod "1ec60992-5bf6-426a-b362-f627124db07c" (UID: "1ec60992-5bf6-426a-b362-f627124db07c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:58.871722 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.871690 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:58.873012 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.872982 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh" pod "1ec60992-5bf6-426a-b362-f627124db07c" (UID: "1ec60992-5bf6-426a-b362-f627124db07c"). InnerVolumeSpecName "kube-api-access-dbdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:58.873221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.873196 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca" pod "1ec60992-5bf6-426a-b362-f627124db07c" (UID: "1ec60992-5bf6-426a-b362-f627124db07c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:58.972948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.972912 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec60992-5bf6-426a-b362-f627124db07c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:58.972948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:58.972943 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbdhh\" (UniqueName: \"kubernetes.io/projected/1ec60992-5bf6-426a-b362-f627124db07c-kube-api-access-dbdhh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:59.186683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.186298 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:57:59.187559 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.187532 5036 patch_prober.go:29] interesting pod/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:57:59.187693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.187601 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:57:59.541347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.531357 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" Oct 30 21:57:59.541347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.531358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6" event={"ID":"1ec60992-5bf6-426a-b362-f627124db07c","Type":"ContainerDied","Data":"96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3"} Oct 30 21:57:59.541347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:57:59.531486 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f1eac1cac58e202edd8097353c1f3606e282ad489e666289892d5d0eb626b3" Oct 30 21:58:00.186826 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:00.186793 5036 patch_prober.go:29] interesting pod/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:58:00.186997 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:00.186851 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:58:00.652913 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:00.652879 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:58:00.653398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:00.653127 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" containerID="cri-o://14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3" gracePeriod=30 Oct 30 21:58:01.155828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.155795 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5"] Oct 30 21:58:01.157070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.157046 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1ec60992-5bf6-426a-b362-f627124db07c" containerName="revoke-keys" Oct 30 21:58:01.157251 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.157234 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec60992-5bf6-426a-b362-f627124db07c" containerName="revoke-keys" Oct 30 21:58:01.157674 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.157652 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec60992-5bf6-426a-b362-f627124db07c" containerName="revoke-keys" Oct 30 21:58:01.162648 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.162630 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.175018 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.174992 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:01.175167 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.174992 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:01.176813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.176776 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:58:01.176933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.176905 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:01.181672 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.181651 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5"] Oct 30 21:58:01.186439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.186415 5036 patch_prober.go:29] interesting pod/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:58:01.186531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.186466 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:58:01.240297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.231335 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pnp\" (UniqueName: \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.240297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.231413 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.240297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.231494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.341292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.339194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pnp\" (UniqueName: \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.341292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.339290 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.341292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.339365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.341292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.340342 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.345071 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.345044 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.361445 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.361421 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pnp\" (UniqueName: \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\") pod \"maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.477736 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.477650 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:01.853923 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:01.853889 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f818703_b73d_4758_a8b2_67ae4dd3cd7a.slice/crio-3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c WatchSource:0}: Error finding container 3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c: Status 404 returned error can't find the container with id 3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c Oct 30 21:58:01.855955 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:01.855930 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5"] Oct 30 21:58:02.197297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:02.194720 5036 patch_prober.go:29] interesting pod/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:58:02.197297 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:02.194783 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:58:02.576591 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:02.576512 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" event={"ID":"6f818703-b73d-4758-a8b2-67ae4dd3cd7a","Type":"ContainerStarted","Data":"167144291d3229c9b95c273aae27316b3d34032ed5aca1d767ab16089e7dea77"} Oct 30 21:58:02.576591 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:02.576569 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" event={"ID":"6f818703-b73d-4758-a8b2-67ae4dd3cd7a","Type":"ContainerStarted","Data":"3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c"} Oct 30 21:58:02.597520 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:02.597470 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" podStartSLOduration=1.597456095 podStartE2EDuration="1.597456095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:02.595010773 +0000 UTC m=+1106.544965719" watchObservedRunningTime="2026-10-30 21:58:02.597456095 +0000 UTC m=+1106.547411041" Oct 30 21:58:03.166978 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:03.166943 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f818703_b73d_4758_a8b2_67ae4dd3cd7a.slice/crio-conmon-167144291d3229c9b95c273aae27316b3d34032ed5aca1d767ab16089e7dea77.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:58:03.186469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:03.186441 5036 patch_prober.go:29] interesting pod/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:58:03.186590 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:03.186485 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:58:03.587222 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:03.587181 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f818703-b73d-4758-a8b2-67ae4dd3cd7a" containerID="167144291d3229c9b95c273aae27316b3d34032ed5aca1d767ab16089e7dea77" exitCode=0 Oct 30 21:58:03.587480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:03.587435 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" event={"ID":"6f818703-b73d-4758-a8b2-67ae4dd3cd7a","Type":"ContainerDied","Data":"167144291d3229c9b95c273aae27316b3d34032ed5aca1d767ab16089e7dea77"} Oct 30 21:58:03.723733 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:03.721703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:58:04.191181 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.191148 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:58:04.597419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.597351 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:58:04.839317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.839252 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:04.980831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.980797 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\") pod \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " Oct 30 21:58:04.980988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.980901 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\" (UniqueName: \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\") pod \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " Oct 30 21:58:04.980988 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.980955 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\") pod \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\" (UID: \"6f818703-b73d-4758-a8b2-67ae4dd3cd7a\") " Oct 30 21:58:04.981373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.981340 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca" pod "6f818703-b73d-4758-a8b2-67ae4dd3cd7a" (UID: "6f818703-b73d-4758-a8b2-67ae4dd3cd7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:04.981647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.981588 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.983082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.983058 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca" pod "6f818703-b73d-4758-a8b2-67ae4dd3cd7a" (UID: "6f818703-b73d-4758-a8b2-67ae4dd3cd7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:04.983185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:04.983138 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp" pod "6f818703-b73d-4758-a8b2-67ae4dd3cd7a" (UID: "6f818703-b73d-4758-a8b2-67ae4dd3cd7a"). InnerVolumeSpecName "kube-api-access-v4pnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:05.083693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.083660 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:05.083693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.083695 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pnp\" (UniqueName: \"kubernetes.io/projected/6f818703-b73d-4758-a8b2-67ae4dd3cd7a-kube-api-access-v4pnp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:05.613628 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.613584 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" Oct 30 21:58:05.614214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.613636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5" event={"ID":"6f818703-b73d-4758-a8b2-67ae4dd3cd7a","Type":"ContainerDied","Data":"3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c"} Oct 30 21:58:05.614214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.613680 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7d0e84ceb45a6609bc5ec25762cd03ee66d9ba4d8b625daa0508859f22601c" Oct 30 21:58:05.741984 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.741948 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:58:05.743358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.743330 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f818703-b73d-4758-a8b2-67ae4dd3cd7a" containerName="revoke-keys" Oct 30 21:58:05.743523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.743501 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f818703-b73d-4758-a8b2-67ae4dd3cd7a" containerName="revoke-keys" Oct 30 21:58:05.743724 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.743711 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f818703-b73d-4758-a8b2-67ae4dd3cd7a" containerName="revoke-keys" Oct 30 21:58:05.748425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.748402 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.760973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.760944 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-004c17-openshift-default-dockercfg-9wc5d\"" Oct 30 21:58:05.767933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.767909 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.902734 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.902852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.902882 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.902958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.903021 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.903121 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.903154 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.903184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gv47\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:05.918364 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:05.903217 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.102051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.102005 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.120505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.120470 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.123889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.123862 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.124565 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.124534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.126607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.121090 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.127578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.124402 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.129340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.129319 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.130834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.130722 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.132512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.132491 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.133187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.130250 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.133686 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.131957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.133957 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.133149 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv47\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.134120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.134106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.135469 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.135451 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.153502 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.152835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.163131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.157245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.163131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.157947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.163131 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.163011 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv47\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\") pod \"e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.241664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.241632 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:58:06.353929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.348779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.358575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.358541 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.362137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.362095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.362655 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.362619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.362911 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.362896 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.363634 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.363465 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.364215 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.364181 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.364399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.364380 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.364717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.364689 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\") pod \"23b880d4-a670-414e-a0fe-f09c66e92f3d\" (UID: \"23b880d4-a670-414e-a0fe-f09c66e92f3d\") " Oct 30 21:58:06.369535 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.369453 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:06.371436 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.371411 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.371723 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.371690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.373509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.372827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:06.374596 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.374337 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.375449 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.375425 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.375735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.375715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.376210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.376124 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:06.390296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.389134 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:06.407725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.407678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v" pod "23b880d4-a670-414e-a0fe-f09c66e92f3d" (UID: "23b880d4-a670-414e-a0fe-f09c66e92f3d"). InnerVolumeSpecName "kube-api-access-bpz7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:06.469010 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.468985 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469010 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469010 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469020 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469029 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469038 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469056 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpz7v\" (UniqueName: \"kubernetes.io/projected/23b880d4-a670-414e-a0fe-f09c66e92f3d-kube-api-access-bpz7v\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469065 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469073 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b880d4-a670-414e-a0fe-f09c66e92f3d-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.469196 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.469081 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b880d4-a670-414e-a0fe-f09c66e92f3d-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.626201 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.626163 5036 generic.go:356] "Generic (PLEG): container finished" podID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerID="14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3" exitCode=0 Oct 30 21:58:06.626646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.626226 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" Oct 30 21:58:06.626646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.626255 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerDied","Data":"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3"} Oct 30 21:58:06.626646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.626371 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46" event={"ID":"23b880d4-a670-414e-a0fe-f09c66e92f3d","Type":"ContainerDied","Data":"c6699fd37ac5384654dbedc433236eeef39f6cd64912c7ac0915ea76463fe4ef"} Oct 30 21:58:06.626646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.626407 5036 scope.go:122] "RemoveContainer" containerID="14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3" Oct 30 21:58:06.641213 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.641186 5036 scope.go:122] "RemoveContainer" containerID="e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7" Oct 30 21:58:06.656772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.656742 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:58:06.657354 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.657330 5036 scope.go:122] "RemoveContainer" containerID="14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3" Oct 30 21:58:06.657740 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:06.657712 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3\": container with ID starting with 14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3 not found: ID does not exist" containerID="14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3" Oct 30 21:58:06.657831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.657751 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3"} err="failed to get container status \"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3\": rpc error: code = NotFound desc = could not find container \"14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3\": container with ID starting with 14dd4ee902348f49d320cd83bc9ecebca4ad14ab046fcd4401583e8fdfcda9c3 not found: ID does not exist" Oct 30 21:58:06.657831 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.657779 5036 scope.go:122] "RemoveContainer" containerID="e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7" Oct 30 21:58:06.658121 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:06.658090 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7\": container with ID starting with e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7 not found: ID does not exist" containerID="e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7" Oct 30 21:58:06.658242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.658128 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7"} err="failed to get container status \"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7\": rpc error: code = NotFound desc = could not find container \"e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7\": container with ID starting with e9eba6135be57dd7cef78e53c17af7d94353393a40d6b4c958e70b0419e3eee7 not found: ID does not exist" Oct 30 21:58:06.662430 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.662409 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-03f0a0-openshift-default-656855c79c-fgm46"] Oct 30 21:58:06.695426 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:06.695399 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda316ac0c_a489_466e_a974_40530fa15cae.slice/crio-176398b9bb4c3e62532a7c78340194be84dd197103fc6bfc2ae6330fd2573480 WatchSource:0}: Error finding container 176398b9bb4c3e62532a7c78340194be84dd197103fc6bfc2ae6330fd2573480: Status 404 returned error can't find the container with id 176398b9bb4c3e62532a7c78340194be84dd197103fc6bfc2ae6330fd2573480 Oct 30 21:58:06.695563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.695547 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:58:06.697868 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.697836 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:06.697974 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.697911 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:06.697974 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:06.697958 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:07.640629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:07.640594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerStarted","Data":"de8dc85eb7fc4ceaaeeb97e10b10c1f4a98ed796122ee5278d6be3acca77d408"} Oct 30 21:58:07.641008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:07.640690 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerStarted","Data":"176398b9bb4c3e62532a7c78340194be84dd197103fc6bfc2ae6330fd2573480"} Oct 30 21:58:07.669132 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:07.669086 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podStartSLOduration=2.6690724279999998 podStartE2EDuration="2.669072428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:07.663736352 +0000 UTC m=+1111.613691320" watchObservedRunningTime="2026-10-30 21:58:07.669072428 +0000 UTC m=+1111.619027373" Oct 30 21:58:08.190444 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:08.190403 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" path="/var/lib/kubelet/pods/23b880d4-a670-414e-a0fe-f09c66e92f3d/volumes" Oct 30 21:58:08.373805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:08.373773 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:08.379311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:08.379283 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:08.652404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:08.652375 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:08.653348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:08.653324 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:58:11.741560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:11.738927 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:58:12.243294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.239084 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:58:12.465294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.462394 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:12.492501 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.480741 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:12.600423 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.600329 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:58:12.658292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.657427 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 21:58:12.770418 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.770326 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj"] Oct 30 21:58:12.843293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:12.813775 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3dfc762-nzgkj"] Oct 30 21:58:13.091109 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.091071 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.092381 5036 cpu_manager.go:414] "containerMap: removing container" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.102698 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.102891 5036 cpu_manager.go:414] "containerMap: removing container" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.102901 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.103201 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.105871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.103216 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b880d4-a670-414e-a0fe-f09c66e92f3d" containerName="istio-proxy" Oct 30 21:58:13.135507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.135474 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:58:13.135938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.135800 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.224311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.168760 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-004c17\"" Oct 30 21:58:13.224311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.194616 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:58:13.224311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.194664 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:58:13.224311 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.196122 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.250291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.225013 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-004c17-dockercfg-jl9db\"" Oct 30 21:58:13.471480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.471450 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xgh\" (UniqueName: \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.471888 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.471862 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.472084 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.472065 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n68h\" (UniqueName: \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.472185 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.472149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.573425 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.573391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.573626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.573475 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n68h\" (UniqueName: \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.573626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.573515 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.573626 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.573581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xgh\" (UniqueName: \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.574229 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.574207 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.574404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.574385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.585959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.585928 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n68h\" (UniqueName: \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\") pod \"payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:13.587186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.587161 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xgh\" (UniqueName: \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\") pod \"payload-processing-e2e-api-a-004c17-55b855d645-klk2s\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.704639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.704560 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:13.759782 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:13.759759 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:14.107107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.106901 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:58:14.137064 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.137033 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:58:14.154554 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:14.154527 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ea4e15_20b4_495b_926c_5c9348176e44.slice/crio-3bd17530ad710398f759250c9bf977f7b28b650810b8881a202675b165084719 WatchSource:0}: Error finding container 3bd17530ad710398f759250c9bf977f7b28b650810b8881a202675b165084719: Status 404 returned error can't find the container with id 3bd17530ad710398f759250c9bf977f7b28b650810b8881a202675b165084719 Oct 30 21:58:14.208688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.208656 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74f9f78-2df3-4358-9ec7-66cfd3f05287" path="/var/lib/kubelet/pods/d74f9f78-2df3-4358-9ec7-66cfd3f05287/volumes" Oct 30 21:58:14.563824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.563784 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" event={"ID":"f0ea4e15-20b4-495b-926c-5c9348176e44","Type":"ContainerStarted","Data":"faf94fc673715a1bc75ce9b7f6fd20def0f9c0bc6f2378ccb8aa2dc04c1c6b59"} Oct 30 21:58:14.564045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.563855 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:14.564045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.563884 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" event={"ID":"f0ea4e15-20b4-495b-926c-5c9348176e44","Type":"ContainerStarted","Data":"3bd17530ad710398f759250c9bf977f7b28b650810b8881a202675b165084719"} Oct 30 21:58:14.565689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.565658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" event={"ID":"ccd4e3ec-5f03-4611-aba9-b394d024359d","Type":"ContainerStarted","Data":"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426"} Oct 30 21:58:14.565824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.565710 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" event={"ID":"ccd4e3ec-5f03-4611-aba9-b394d024359d","Type":"ContainerStarted","Data":"33ca2ee216aed3f467efcc9d6f5f100782bfc116375cb0c0b5433e476f5605c7"} Oct 30 21:58:14.565824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.565769 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:14.583959 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.583915 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" podStartSLOduration=1.583902154 podStartE2EDuration="1.583902154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.582103895 +0000 UTC m=+1118.532058840" watchObservedRunningTime="2026-10-30 21:58:14.583902154 +0000 UTC m=+1118.533857101" Oct 30 21:58:14.602926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:14.602884 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" podStartSLOduration=1.602872438 podStartE2EDuration="1.602872438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.599822057 +0000 UTC m=+1118.549777004" watchObservedRunningTime="2026-10-30 21:58:14.602872438 +0000 UTC m=+1118.552827362" Oct 30 21:58:16.740294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:16.740181 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" podUID="91b64500-c261-45d6-81ed-c465dd6914a0" containerName="payload-processing" containerID="cri-o://67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566" gracePeriod=25 Oct 30 21:58:17.095725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.095697 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:58:17.214915 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.214879 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\") pod \"91b64500-c261-45d6-81ed-c465dd6914a0\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " Oct 30 21:58:17.215144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.214961 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\" (UniqueName: \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\") pod \"91b64500-c261-45d6-81ed-c465dd6914a0\" (UID: \"91b64500-c261-45d6-81ed-c465dd6914a0\") " Oct 30 21:58:17.218159 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.218073 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume" pod "91b64500-c261-45d6-81ed-c465dd6914a0" (UID: "91b64500-c261-45d6-81ed-c465dd6914a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:17.221006 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.220971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln" pod "91b64500-c261-45d6-81ed-c465dd6914a0" (UID: "91b64500-c261-45d6-81ed-c465dd6914a0"). InnerVolumeSpecName "kube-api-access-2g2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:17.318475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.318381 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b64500-c261-45d6-81ed-c465dd6914a0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:17.318475 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.318414 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2ln\" (UniqueName: \"kubernetes.io/projected/91b64500-c261-45d6-81ed-c465dd6914a0-kube-api-access-2g2ln\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:17.601223 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.601118 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" podUID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" containerName="payload-pre-processing" containerID="cri-o://5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7" gracePeriod=25 Oct 30 21:58:17.699765 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.699732 5036 generic.go:356] "Generic (PLEG): container finished" podID="91b64500-c261-45d6-81ed-c465dd6914a0" containerID="67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566" exitCode=0 Oct 30 21:58:17.700278 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.700236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" event={"ID":"91b64500-c261-45d6-81ed-c465dd6914a0","Type":"ContainerDied","Data":"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566"} Oct 30 21:58:17.700462 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.700445 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" event={"ID":"91b64500-c261-45d6-81ed-c465dd6914a0","Type":"ContainerDied","Data":"16dd8fae387294ff93dcbf149ef47907d72106dafc0ca87d92352ef5168488f4"} Oct 30 21:58:17.700600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.700586 5036 scope.go:122] "RemoveContainer" containerID="67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566" Oct 30 21:58:17.700870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.700853 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64" Oct 30 21:58:17.794144 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.794010 5036 scope.go:122] "RemoveContainer" containerID="67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566" Oct 30 21:58:17.795239 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:17.794485 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566\": container with ID starting with 67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566 not found: ID does not exist" containerID="67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566" Oct 30 21:58:17.795239 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.794525 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566"} err="failed to get container status \"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566\": rpc error: code = NotFound desc = could not find container \"67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566\": container with ID starting with 67971ae03a745df7576e1a377d2c9754ecd58d9c31247265b1ff61f968687566 not found: ID does not exist" Oct 30 21:58:17.795883 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.795827 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:58:17.803322 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:17.802433 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56c0af-598dc77555-zkp64"] Oct 30 21:58:18.096120 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.096059 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf"] Oct 30 21:58:18.105197 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.105135 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee0c56-5hbvf"] Oct 30 21:58:18.164766 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.164328 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:58:18.189653 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:18.189612 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:58:18.193749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.193714 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\" (UniqueName: \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\") pod \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " Oct 30 21:58:18.193938 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.193874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\") pod \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\" (UID: \"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb\") " Oct 30 21:58:18.198692 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.198649 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume" pod "c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" (UID: "c80ecc44-9246-4ca0-86b9-6e9abd27f4eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:18.237559 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.237518 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx" pod "c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" (UID: "c80ecc44-9246-4ca0-86b9-6e9abd27f4eb"). InnerVolumeSpecName "kube-api-access-dm5xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:18.243280 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.243238 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b64500-c261-45d6-81ed-c465dd6914a0" path="/var/lib/kubelet/pods/91b64500-c261-45d6-81ed-c465dd6914a0/volumes" Oct 30 21:58:18.244065 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.244042 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c2f7be-1198-4333-a595-6805d27034ad" path="/var/lib/kubelet/pods/d8c2f7be-1198-4333-a595-6805d27034ad/volumes" Oct 30 21:58:18.298871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.298829 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5xx\" (UniqueName: \"kubernetes.io/projected/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-kube-api-access-dm5xx\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:18.298871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.298862 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:18.712697 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.712656 5036 generic.go:356] "Generic (PLEG): container finished" podID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" containerID="5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7" exitCode=0 Oct 30 21:58:18.712882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.712701 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" Oct 30 21:58:18.712882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.712735 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" event={"ID":"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb","Type":"ContainerDied","Data":"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7"} Oct 30 21:58:18.712882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.712801 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m" event={"ID":"c80ecc44-9246-4ca0-86b9-6e9abd27f4eb","Type":"ContainerDied","Data":"e9b2c4a3298f2e407cd98dcd3603493d2c17db429d041c77fd68c2a7d9383e72"} Oct 30 21:58:18.712882 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.712830 5036 scope.go:122] "RemoveContainer" containerID="5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7" Oct 30 21:58:18.753334 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.750150 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:58:18.758909 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.758827 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56c0af-58d4fd4b7-czk9m"] Oct 30 21:58:18.768807 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.768571 5036 scope.go:122] "RemoveContainer" containerID="5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7" Oct 30 21:58:18.769226 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:18.769191 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7\": container with ID starting with 5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7 not found: ID does not exist" containerID="5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7" Oct 30 21:58:18.769458 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:18.769428 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7"} err="failed to get container status \"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7\": rpc error: code = NotFound desc = could not find container \"5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7\": container with ID starting with 5d081a3be536d0efb85ee361389ff86d6033b21ef9734e7d6d43d5bbb1eb1ae7 not found: ID does not exist" Oct 30 21:58:19.031070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:19.030978 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t"] Oct 30 21:58:19.039689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:19.039648 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440b92b3-5vt4t"] Oct 30 21:58:20.190193 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.190158 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" path="/var/lib/kubelet/pods/c80ecc44-9246-4ca0-86b9-6e9abd27f4eb/volumes" Oct 30 21:58:20.190839 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.190818 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbeeeb53-52e6-4d1a-be28-45e4690cec3f" path="/var/lib/kubelet/pods/cbeeeb53-52e6-4d1a-be28-45e4690cec3f/volumes" Oct 30 21:58:20.708422 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.708394 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:20.709090 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709072 5036 cpu_manager.go:414] "containerMap: removing container" podUID="91b64500-c261-45d6-81ed-c465dd6914a0" containerName="payload-processing" Oct 30 21:58:20.709139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709092 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b64500-c261-45d6-81ed-c465dd6914a0" containerName="payload-processing" Oct 30 21:58:20.709139 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709136 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" containerName="payload-pre-processing" Oct 30 21:58:20.709221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709143 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" containerName="payload-pre-processing" Oct 30 21:58:20.709306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709296 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80ecc44-9246-4ca0-86b9-6e9abd27f4eb" containerName="payload-pre-processing" Oct 30 21:58:20.709339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.709312 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b64500-c261-45d6-81ed-c465dd6914a0" containerName="payload-processing" Oct 30 21:58:20.714944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.714920 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.729355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.729331 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:20.730804 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.730767 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3654d5\"" Oct 30 21:58:20.765790 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.765754 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.765960 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.765938 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkb9\" (UniqueName: \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.774161 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.771295 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:20.777609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.777574 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:20.784319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.782987 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:20.796385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.796352 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3654d5-dockercfg-4tfxz\"" Oct 30 21:58:20.891289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.890517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.891289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.890651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5v7\" (UniqueName: \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:20.891289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.890743 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:20.891289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.890832 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkb9\" (UniqueName: \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.904310 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.892240 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.929828 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.929761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkb9\" (UniqueName: \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:20.995360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.993285 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5v7\" (UniqueName: \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:20.995360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.993392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:20.995360 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:20.994602 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:21.008352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.007512 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5v7\" (UniqueName: \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\") pod \"payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:21.040732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.040350 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:21.099939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.099730 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:21.394377 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:21.394347 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969d71c0_f8cb_41e6_a677_4c2cd9bc63b0.slice/crio-a855710d6a28499e052e03afb0eab193e4f0cac3f14b31aaf901c808b90c511a WatchSource:0}: Error finding container a855710d6a28499e052e03afb0eab193e4f0cac3f14b31aaf901c808b90c511a: Status 404 returned error can't find the container with id a855710d6a28499e052e03afb0eab193e4f0cac3f14b31aaf901c808b90c511a Oct 30 21:58:21.395128 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.395105 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:21.487601 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:21.487576 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf740fe_410e_48e5_9ead_f25bdbfb2954.slice/crio-6be39d9e8c134f05a50b0e227e9b6117a70a86403ff93aa0f1ebd8bc92199e52 WatchSource:0}: Error finding container 6be39d9e8c134f05a50b0e227e9b6117a70a86403ff93aa0f1ebd8bc92199e52: Status 404 returned error can't find the container with id 6be39d9e8c134f05a50b0e227e9b6117a70a86403ff93aa0f1ebd8bc92199e52 Oct 30 21:58:21.488780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.488750 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:21.759927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.759824 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" event={"ID":"eaf740fe-410e-48e5-9ead-f25bdbfb2954","Type":"ContainerStarted","Data":"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f"} Oct 30 21:58:21.759927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.759876 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" event={"ID":"eaf740fe-410e-48e5-9ead-f25bdbfb2954","Type":"ContainerStarted","Data":"6be39d9e8c134f05a50b0e227e9b6117a70a86403ff93aa0f1ebd8bc92199e52"} Oct 30 21:58:21.759927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.759903 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:21.761731 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.761702 5036 generic.go:356] "Generic (PLEG): container finished" podID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerID="432c3d6ee7ac53057ed9d918a7eb60908f4a86c46500b203e074554f572aa93f" exitCode=0 Oct 30 21:58:21.761884 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.761774 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerDied","Data":"432c3d6ee7ac53057ed9d918a7eb60908f4a86c46500b203e074554f572aa93f"} Oct 30 21:58:21.762176 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.762162 5036 scope.go:122] "RemoveContainer" containerID="432c3d6ee7ac53057ed9d918a7eb60908f4a86c46500b203e074554f572aa93f" Oct 30 21:58:21.763636 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763613 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" event={"ID":"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0","Type":"ContainerStarted","Data":"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf"} Oct 30 21:58:21.763725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763660 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" event={"ID":"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0","Type":"ContainerStarted","Data":"a855710d6a28499e052e03afb0eab193e4f0cac3f14b31aaf901c808b90c511a"} Oct 30 21:58:21.763725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763683 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:21.763844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763794 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:21.763875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763858 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:21.763908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.763891 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:21.783973 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.783932 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" podStartSLOduration=1.7839193789999999 podStartE2EDuration="1.783919379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:21.778947803 +0000 UTC m=+1125.728902753" watchObservedRunningTime="2026-10-30 21:58:21.783919379 +0000 UTC m=+1125.733874335" Oct 30 21:58:21.827341 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:21.820238 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" podStartSLOduration=1.820219963 podStartE2EDuration="1.820219963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:21.816932368 +0000 UTC m=+1125.766887315" watchObservedRunningTime="2026-10-30 21:58:21.820219963 +0000 UTC m=+1125.770174910" Oct 30 21:58:22.421320 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.421285 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5"] Oct 30 21:58:22.454843 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.454797 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56c0af-wtxc5"] Oct 30 21:58:22.775735 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.775650 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerStarted","Data":"160634d683a6a5af06cf5f027bd72bda14eb4e6e4278bec7db3cf984b64c7881"} Oct 30 21:58:22.941290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.940699 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:58:22.941290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.940828 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:22.941290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:22.940874 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:23.047808 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:23.047727 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf"] Oct 30 21:58:23.055919 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:23.055889 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35678ea2-dvzgf"] Oct 30 21:58:23.939291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:23.935619 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:23.939291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:23.935676 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:24.190815 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.190742 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce2f42c-0d56-4e61-b04f-01a993b10388" path="/var/lib/kubelet/pods/4ce2f42c-0d56-4e61-b04f-01a993b10388/volumes" Oct 30 21:58:24.191220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.191206 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f818703-b73d-4758-a8b2-67ae4dd3cd7a" path="/var/lib/kubelet/pods/6f818703-b73d-4758-a8b2-67ae4dd3cd7a/volumes" Oct 30 21:58:24.191568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.191555 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m"] Oct 30 21:58:24.200605 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.200580 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efe13b0-b5q9m"] Oct 30 21:58:24.936725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.936686 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:24.937046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:24.937016 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:25.695337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:25.690544 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:58:25.743295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:25.737586 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:58:25.969361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:25.961841 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:25.969361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:25.961909 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:26.190086 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:26.190056 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5661e199-85a5-4a15-9ad5-720233255d95" path="/var/lib/kubelet/pods/5661e199-85a5-4a15-9ad5-720233255d95/volumes" Oct 30 21:58:26.942308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:26.941182 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:26.942308 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:26.941258 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:27.942294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:27.939628 5036 patch_prober.go:29] interesting pod/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:58:27.942294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:27.939701 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:58:28.276921 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.276829 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:58:28.277142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.277114 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" containerID="cri-o://4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1" gracePeriod=30 Oct 30 21:58:28.790900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.790862 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm"] Oct 30 21:58:28.797231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.797206 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.800920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.800893 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:28.801066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.800959 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:58:28.802008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.801477 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:28.802008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.801678 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:28.807130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.807103 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm"] Oct 30 21:58:28.820751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.820719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.821113 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.821082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.821244 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.821220 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqz4\" (UniqueName: \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.922251 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.922207 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.922480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.922342 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.922480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.922445 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqz4\" (UniqueName: \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.923102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.923081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.925066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.925031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.937439 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.937408 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqz4\" (UniqueName: \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\") pod \"maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:28.939880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.939857 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:58:28.940379 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.940360 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:58:28.941138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:28.941122 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:58:29.127244 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:29.126355 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:29.423812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:29.423778 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm"] Oct 30 21:58:29.424479 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:29.424449 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba85e63_e14f_4bac_ba05_f791ae093f37.slice/crio-a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a WatchSource:0}: Error finding container a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a: Status 404 returned error can't find the container with id a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a Oct 30 21:58:30.121552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:30.121475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" event={"ID":"5ba85e63-e14f-4bac-ba05-f791ae093f37","Type":"ContainerStarted","Data":"f097631355c180a422c315eeb0d7e26b83771a2bb276ea59afea90056e50ab73"} Oct 30 21:58:30.121552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:30.121526 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" event={"ID":"5ba85e63-e14f-4bac-ba05-f791ae093f37","Type":"ContainerStarted","Data":"a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a"} Oct 30 21:58:30.143466 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:30.143420 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" podStartSLOduration=2.143407286 podStartE2EDuration="2.143407286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:30.138600186 +0000 UTC m=+1134.088555153" watchObservedRunningTime="2026-10-30 21:58:30.143407286 +0000 UTC m=+1134.093362231" Oct 30 21:58:31.136877 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:31.136838 5036 generic.go:356] "Generic (PLEG): container finished" podID="5ba85e63-e14f-4bac-ba05-f791ae093f37" containerID="f097631355c180a422c315eeb0d7e26b83771a2bb276ea59afea90056e50ab73" exitCode=0 Oct 30 21:58:31.137087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:31.136901 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" event={"ID":"5ba85e63-e14f-4bac-ba05-f791ae093f37","Type":"ContainerDied","Data":"f097631355c180a422c315eeb0d7e26b83771a2bb276ea59afea90056e50ab73"} Oct 30 21:58:32.372681 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.372653 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:32.461335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.461229 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\" (UniqueName: \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\") pod \"5ba85e63-e14f-4bac-ba05-f791ae093f37\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " Oct 30 21:58:32.461512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.461385 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\") pod \"5ba85e63-e14f-4bac-ba05-f791ae093f37\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " Oct 30 21:58:32.461512 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.461491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\") pod \"5ba85e63-e14f-4bac-ba05-f791ae093f37\" (UID: \"5ba85e63-e14f-4bac-ba05-f791ae093f37\") " Oct 30 21:58:32.461794 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.461769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca" pod "5ba85e63-e14f-4bac-ba05-f791ae093f37" (UID: "5ba85e63-e14f-4bac-ba05-f791ae093f37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:32.462285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.462250 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:32.464053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.464024 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca" pod "5ba85e63-e14f-4bac-ba05-f791ae093f37" (UID: "5ba85e63-e14f-4bac-ba05-f791ae093f37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:32.464385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.464362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4" pod "5ba85e63-e14f-4bac-ba05-f791ae093f37" (UID: "5ba85e63-e14f-4bac-ba05-f791ae093f37"). InnerVolumeSpecName "kube-api-access-rkqz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:32.564184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.564148 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba85e63-e14f-4bac-ba05-f791ae093f37-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:32.564184 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.564186 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkqz4\" (UniqueName: \"kubernetes.io/projected/5ba85e63-e14f-4bac-ba05-f791ae093f37-kube-api-access-rkqz4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:32.783336 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.779972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:32.784401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:32.784378 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:33.052345 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.051408 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb"] Oct 30 21:58:33.067540 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.067503 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66328687-g7blb"] Oct 30 21:58:33.169152 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.169112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" event={"ID":"5ba85e63-e14f-4bac-ba05-f791ae093f37","Type":"ContainerDied","Data":"a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a"} Oct 30 21:58:33.169374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.169178 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97af54340fafe9101be2edf45f4ca15c910c4036ab1e93666580791ad898c2a" Oct 30 21:58:33.169374 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.169256 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm" Oct 30 21:58:33.609399 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.609367 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:58:33.699210 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699257 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699315 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699354 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699492 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699641 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:33.699761 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699720 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699767 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699804 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.699929 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.699902 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\") pod \"7f2f5b07-82d9-487f-b388-405dd9403c91\" (UID: \"7f2f5b07-82d9-487f-b388-405dd9403c91\") " Oct 30 21:58:33.700030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.700005 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.700441 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.700881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.700932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.701064 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f5b07-82d9-487f-b388-405dd9403c91-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.701090 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.701107 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.703607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.703570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:33.716352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.705575 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:33.716352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.705865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "kube-api-access-r9rkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:33.716352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.714981 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo" pod "7f2f5b07-82d9-487f-b388-405dd9403c91" (UID: "7f2f5b07-82d9-487f-b388-405dd9403c91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:33.802356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802321 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.802356 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802349 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-kube-api-access-r9rkn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.802548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802363 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.802548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802377 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f5b07-82d9-487f-b388-405dd9403c91-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.802548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802389 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.802548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:33.802401 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f5b07-82d9-487f-b388-405dd9403c91-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:34.215081 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.215046 5036 generic.go:356] "Generic (PLEG): container finished" podID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerID="4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1" exitCode=0 Oct 30 21:58:34.215531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.215393 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.254210 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591cdea1-d8b7-47ae-b262-1850c5966742" path="/var/lib/kubelet/pods/591cdea1-d8b7-47ae-b262-1850c5966742/volumes" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.255074 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256135 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5ba85e63-e14f-4bac-ba05-f791ae093f37" containerName="revoke-keys" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256155 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba85e63-e14f-4bac-ba05-f791ae093f37" containerName="revoke-keys" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256200 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256207 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256234 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256242 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256441 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba85e63-e14f-4bac-ba05-f791ae093f37" containerName="revoke-keys" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256462 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.257556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.256476 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" containerName="istio-proxy" Oct 30 21:58:34.274328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.274251 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerDied","Data":"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1"} Oct 30 21:58:34.274548 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.274526 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq" event={"ID":"7f2f5b07-82d9-487f-b388-405dd9403c91","Type":"ContainerDied","Data":"9a31f08d68fcbdea39a66dc5538557907c227fe50beaf17cdd1ac64d2029ba67"} Oct 30 21:58:34.274684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.274670 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 21:58:34.275027 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.275008 5036 scope.go:122] "RemoveContainer" containerID="4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1" Oct 30 21:58:34.275612 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.274416 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.287114 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.284518 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0b3fe4-openshift-default-dockercfg-ftffx\"" Oct 30 21:58:34.298698 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.298674 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:58:34.313677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.313399 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373411 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373473 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cl8\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373506 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373668 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.373875 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373830 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.374019 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.373998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.374087 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.374074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.374352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.359338 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee0c56-openshift-default-65b9b49db9-jh4jq"] Oct 30 21:58:34.433317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.431433 5036 scope.go:122] "RemoveContainer" containerID="91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284" Oct 30 21:58:34.461559 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.461537 5036 scope.go:122] "RemoveContainer" containerID="4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1" Oct 30 21:58:34.462211 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:34.462183 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1\": container with ID starting with 4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1 not found: ID does not exist" containerID="4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1" Oct 30 21:58:34.462381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.462223 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1"} err="failed to get container status \"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1\": rpc error: code = NotFound desc = could not find container \"4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1\": container with ID starting with 4d93b9e407c32950cba078b23593f158639df879a7dd363163b8cc7b7d3ccea1 not found: ID does not exist" Oct 30 21:58:34.462381 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.462245 5036 scope.go:122] "RemoveContainer" containerID="91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284" Oct 30 21:58:34.462596 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:34.462576 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284\": container with ID starting with 91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284 not found: ID does not exist" containerID="91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284" Oct 30 21:58:34.462676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.462605 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284"} err="failed to get container status \"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284\": rpc error: code = NotFound desc = could not find container \"91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284\": container with ID starting with 91e0a7ab9b33fa53f6b815e87aa90425b49d580485e42ee9cf115649aec4c284 not found: ID does not exist" Oct 30 21:58:34.475175 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475120 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475175 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475405 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475243 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475405 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475379 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475410 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475474 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475524 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475532 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cl8\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475611 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.475824 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475751 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.476371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475883 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.476371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.475971 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.476371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.476031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.478485 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.478461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.478660 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.478641 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.494533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.494503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cl8\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.504170 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.504147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\") pod \"e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:34.712920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:34.712796 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:35.039402 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.039373 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw"] Oct 30 21:58:35.040135 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:35.040106 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbb1593_f067_4962_af30_67e3758a305d.slice/crio-dd8e86745354f653ba8cd46e250216710e11c3fb3a2a56433867412047d9b0c0 WatchSource:0}: Error finding container dd8e86745354f653ba8cd46e250216710e11c3fb3a2a56433867412047d9b0c0: Status 404 returned error can't find the container with id dd8e86745354f653ba8cd46e250216710e11c3fb3a2a56433867412047d9b0c0 Oct 30 21:58:35.043694 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.043650 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:35.043791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.043735 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:35.043791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.043775 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:35.045869 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.045845 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 21:58:35.052202 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.052177 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d8a522-w4jcw"] Oct 30 21:58:35.232445 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.232344 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerStarted","Data":"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f"} Oct 30 21:58:35.232445 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.232409 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerStarted","Data":"dd8e86745354f653ba8cd46e250216710e11c3fb3a2a56433867412047d9b0c0"} Oct 30 21:58:35.262000 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.261954 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podStartSLOduration=1.261940452 podStartE2EDuration="1.261940452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:35.257709606 +0000 UTC m=+1139.207664553" watchObservedRunningTime="2026-10-30 21:58:35.261940452 +0000 UTC m=+1139.211895399" Oct 30 21:58:35.715279 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:35.715234 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:36.204128 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.204088 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6078ee51-072e-4815-b65c-60b05f4cf6bd" path="/var/lib/kubelet/pods/6078ee51-072e-4815-b65c-60b05f4cf6bd/volumes" Oct 30 21:58:36.205329 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.205252 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2f5b07-82d9-487f-b388-405dd9403c91" path="/var/lib/kubelet/pods/7f2f5b07-82d9-487f-b388-405dd9403c91/volumes" Oct 30 21:58:36.695177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.695143 5036 scope.go:122] "RemoveContainer" containerID="5f661fe8a26d79e8e765a028b0d1adf1726baa3d38cb1746d2919458446763b2" Oct 30 21:58:36.707864 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.707844 5036 scope.go:122] "RemoveContainer" containerID="12deaa2cb6c595c5a170daa8aeabf6979dbd0245bfc668dac6cb0bc8bd970c83" Oct 30 21:58:36.719588 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.719467 5036 scope.go:122] "RemoveContainer" containerID="ccf87d9b962cecd9ce924c41fb47c4ed5a2a0b76948cb2a41831667142a2f01c" Oct 30 21:58:36.722743 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.722721 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:36.730975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.730956 5036 scope.go:122] "RemoveContainer" containerID="c2a8f58c9edfbcce6e8d83df496121c0404993d3cb5651277b54161b07b87cf7" Oct 30 21:58:36.747851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.747829 5036 scope.go:122] "RemoveContainer" containerID="b9a30e457912ce8b7c2909e32784c7116304b8b55a754d7e5877620a89da53d5" Oct 30 21:58:36.758255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.758232 5036 scope.go:122] "RemoveContainer" containerID="61e122adff745990685646b140db9075607c2a9e3371b09be8bcb2c4306d26f7" Oct 30 21:58:36.768799 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:36.768778 5036 scope.go:122] "RemoveContainer" containerID="8d9549edabfe93d4e90fd65d61aacc61f9948ea8fdacc066116d4ca6fa9846de" Oct 30 21:58:37.323509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.323481 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:37.324563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.324539 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:58:37.808864 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.808835 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:58:37.816235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.816212 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.831137 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.831073 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f47f79\"" Oct 30 21:58:37.837348 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.837324 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f47f79-dockercfg-plps2\"" Oct 30 21:58:37.842690 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.842657 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:58:37.848832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.848812 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:58:37.848963 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.848937 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.852856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.852827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7h7z\" (UniqueName: \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.852998 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.852891 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.853049 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.852999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.853085 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.853071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265lb\" (UniqueName: \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.882333 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.878081 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:58:37.954174 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.954135 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.954357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.954227 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265lb\" (UniqueName: \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.954483 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.954467 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7h7z\" (UniqueName: \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.954550 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.954519 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.954940 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.954921 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:37.955453 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.955427 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.984463 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.984441 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7h7z\" (UniqueName: \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\") pod \"payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:37.985701 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:37.985682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265lb\" (UniqueName: \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\") pod \"payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:38.151292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:38.139738 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:38.208097 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:38.207542 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:38.601367 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:38.601338 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb855346_a630_40f7_b32b_91dccdc68205.slice/crio-964326264dba5653e54ef8d2805d3067221bfad401de7e92f916feda8f9c3f02 WatchSource:0}: Error finding container 964326264dba5653e54ef8d2805d3067221bfad401de7e92f916feda8f9c3f02: Status 404 returned error can't find the container with id 964326264dba5653e54ef8d2805d3067221bfad401de7e92f916feda8f9c3f02 Oct 30 21:58:38.602191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:38.602130 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:58:38.623946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:38.623923 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:58:38.624632 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:38.624607 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fec8243_a064_4c9d_920a_7e960ba0c790.slice/crio-b8d46344662e1e30eff415510119af88440bea25c7fcbba30c95a64ee10412dd WatchSource:0}: Error finding container b8d46344662e1e30eff415510119af88440bea25c7fcbba30c95a64ee10412dd: Status 404 returned error can't find the container with id b8d46344662e1e30eff415510119af88440bea25c7fcbba30c95a64ee10412dd Oct 30 21:58:39.420337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.403018 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" event={"ID":"eb855346-a630-40f7-b32b-91dccdc68205","Type":"ContainerStarted","Data":"7b3d7be8363c74c98505c1e27db9174ed27d8cfec9cdfe18a6a0f5e2a8427377"} Oct 30 21:58:39.420337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.403133 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" event={"ID":"eb855346-a630-40f7-b32b-91dccdc68205","Type":"ContainerStarted","Data":"964326264dba5653e54ef8d2805d3067221bfad401de7e92f916feda8f9c3f02"} Oct 30 21:58:39.420337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.404342 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:39.432390 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.432349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" event={"ID":"3fec8243-a064-4c9d-920a-7e960ba0c790","Type":"ContainerStarted","Data":"62afcff1028b1dd1f82d2fc145fb8c098728d2373592055aa0f3ed60538a8464"} Oct 30 21:58:39.432640 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.432620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" event={"ID":"3fec8243-a064-4c9d-920a-7e960ba0c790","Type":"ContainerStarted","Data":"b8d46344662e1e30eff415510119af88440bea25c7fcbba30c95a64ee10412dd"} Oct 30 21:58:39.432760 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.432748 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:39.442189 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.442112 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" podStartSLOduration=2.442094561 podStartE2EDuration="2.442094561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.440850126 +0000 UTC m=+1143.390805073" watchObservedRunningTime="2026-10-30 21:58:39.442094561 +0000 UTC m=+1143.392049510" Oct 30 21:58:39.479166 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:39.479106 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" podStartSLOduration=2.479088025 podStartE2EDuration="2.479088025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.47694417 +0000 UTC m=+1143.426899124" watchObservedRunningTime="2026-10-30 21:58:39.479088025 +0000 UTC m=+1143.429042973" Oct 30 21:58:40.390661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:40.390625 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:40.428180 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:40.428145 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:42.766145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.766113 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 21:58:42.771164 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.771140 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.801902 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.801417 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8d2c42-openshift-default-dockercfg-xdw5s\"" Oct 30 21:58:42.808396 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.808367 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831225 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831490 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6t2x\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831569 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.837321 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.831805 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933425 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933457 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933610 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933654 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6t2x\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933688 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933761 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.933947 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.933913 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.934433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.934407 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.934763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.934739 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.935009 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.934989 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.935224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.935203 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.936149 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.936118 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:42.936648 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:42.936604 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:43.016896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.016776 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6t2x\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:43.017600 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.017573 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\") pod \"e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:43.088857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.088553 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:43.527175 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:43.527139 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45a8d71_db14_47f4_a20e_7f7ebb33d20a.slice/crio-2020c29ab615b4b276d2199f7ffb35a8b81d6abe10baa3a5690ef709a0eae9f2 WatchSource:0}: Error finding container 2020c29ab615b4b276d2199f7ffb35a8b81d6abe10baa3a5690ef709a0eae9f2: Status 404 returned error can't find the container with id 2020c29ab615b4b276d2199f7ffb35a8b81d6abe10baa3a5690ef709a0eae9f2 Oct 30 21:58:43.527316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.527287 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 21:58:43.530138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.530102 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:43.530221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.530169 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:43.530221 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:43.530200 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:44.511044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:44.511010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerStarted","Data":"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f"} Oct 30 21:58:44.511557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:44.511067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerStarted","Data":"2020c29ab615b4b276d2199f7ffb35a8b81d6abe10baa3a5690ef709a0eae9f2"} Oct 30 21:58:44.534787 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:44.534711 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podStartSLOduration=2.534695186 podStartE2EDuration="2.534695186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:44.533691822 +0000 UTC m=+1148.483646768" watchObservedRunningTime="2026-10-30 21:58:44.534695186 +0000 UTC m=+1148.484650133" Oct 30 21:58:45.089679 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.089642 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:45.095555 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.095514 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:45.399296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.391435 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" podUID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" containerName="payload-processing" containerID="cri-o://cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f" gracePeriod=25 Oct 30 21:58:45.457620 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.439925 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" podUID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" containerName="payload-pre-processing" containerID="cri-o://9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf" gracePeriod=25 Oct 30 21:58:45.558235 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.557118 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:45.562533 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:45.562503 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 21:58:46.001226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.001155 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:46.010047 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.010018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\" (UniqueName: \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\") pod \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " Oct 30 21:58:46.010199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.010073 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\") pod \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\" (UID: \"eaf740fe-410e-48e5-9ead-f25bdbfb2954\") " Oct 30 21:58:46.010967 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.010937 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume" pod "eaf740fe-410e-48e5-9ead-f25bdbfb2954" (UID: "eaf740fe-410e-48e5-9ead-f25bdbfb2954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:46.017042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.017012 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7" pod "eaf740fe-410e-48e5-9ead-f25bdbfb2954" (UID: "eaf740fe-410e-48e5-9ead-f25bdbfb2954"). InnerVolumeSpecName "kube-api-access-pk5v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.121314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.120539 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5v7\" (UniqueName: \"kubernetes.io/projected/eaf740fe-410e-48e5-9ead-f25bdbfb2954-kube-api-access-pk5v7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:46.121314 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.120567 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf740fe-410e-48e5-9ead-f25bdbfb2954-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:46.121795 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.121752 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:46.324187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.324101 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\") pod \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " Oct 30 21:58:46.324187 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.324163 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\" (UniqueName: \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\") pod \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\" (UID: \"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0\") " Oct 30 21:58:46.324582 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.324558 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume" pod "969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" (UID: "969d71c0-f8cb-41e6-a677-4c2cd9bc63b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:46.324886 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.324866 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:46.326621 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.326593 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9" pod "969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" (UID: "969d71c0-f8cb-41e6-a677-4c2cd9bc63b0"). InnerVolumeSpecName "kube-api-access-snkb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.426358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.426327 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkb9\" (UniqueName: \"kubernetes.io/projected/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0-kube-api-access-snkb9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:46.570992 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.570946 5036 generic.go:356] "Generic (PLEG): container finished" podID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" containerID="9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf" exitCode=0 Oct 30 21:58:46.571537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.571018 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" event={"ID":"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0","Type":"ContainerDied","Data":"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf"} Oct 30 21:58:46.571537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.571087 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" event={"ID":"969d71c0-f8cb-41e6-a677-4c2cd9bc63b0","Type":"ContainerDied","Data":"a855710d6a28499e052e03afb0eab193e4f0cac3f14b31aaf901c808b90c511a"} Oct 30 21:58:46.571537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.571138 5036 scope.go:122] "RemoveContainer" containerID="9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf" Oct 30 21:58:46.571537 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.571139 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk" Oct 30 21:58:46.573388 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.573348 5036 generic.go:356] "Generic (PLEG): container finished" podID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" containerID="cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f" exitCode=0 Oct 30 21:58:46.574463 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.574412 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" Oct 30 21:58:46.575599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.575491 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" event={"ID":"eaf740fe-410e-48e5-9ead-f25bdbfb2954","Type":"ContainerDied","Data":"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f"} Oct 30 21:58:46.575599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.575540 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g" event={"ID":"eaf740fe-410e-48e5-9ead-f25bdbfb2954","Type":"ContainerDied","Data":"6be39d9e8c134f05a50b0e227e9b6117a70a86403ff93aa0f1ebd8bc92199e52"} Oct 30 21:58:46.589168 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.589146 5036 scope.go:122] "RemoveContainer" containerID="9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf" Oct 30 21:58:46.589562 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:46.589540 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf\": container with ID starting with 9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf not found: ID does not exist" containerID="9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf" Oct 30 21:58:46.589666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.589568 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf"} err="failed to get container status \"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf\": rpc error: code = NotFound desc = could not find container \"9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf\": container with ID starting with 9bdb624e69903700472ac507fec015ca8fc06f84370d44496136cc03a19eaebf not found: ID does not exist" Oct 30 21:58:46.589666 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.589585 5036 scope.go:122] "RemoveContainer" containerID="cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f" Oct 30 21:58:46.603891 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.603869 5036 scope.go:122] "RemoveContainer" containerID="cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f" Oct 30 21:58:46.604278 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:46.604238 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f\": container with ID starting with cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f not found: ID does not exist" containerID="cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f" Oct 30 21:58:46.604383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.604292 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f"} err="failed to get container status \"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f\": rpc error: code = NotFound desc = could not find container \"cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f\": container with ID starting with cb26567f354c53ef469cb65c8557d2c38fbc762aadf8841f2e82c343d1f3c54f not found: ID does not exist" Oct 30 21:58:46.632116 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.632085 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:46.648574 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.648547 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-586bdff7f9-t5m5g"] Oct 30 21:58:46.655004 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.654977 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:46.660136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.660110 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-7dc4f4c67-ln9fk"] Oct 30 21:58:46.750530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.750478 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:58:46.751638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751611 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" containerName="payload-processing" Oct 30 21:58:46.751638 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751638 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" containerName="payload-processing" Oct 30 21:58:46.751889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751671 5036 cpu_manager.go:414] "containerMap: removing container" podUID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" containerName="payload-pre-processing" Oct 30 21:58:46.751889 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751682 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" containerName="payload-pre-processing" Oct 30 21:58:46.752053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751913 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" containerName="payload-processing" Oct 30 21:58:46.752053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.751932 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" containerName="payload-pre-processing" Oct 30 21:58:46.760111 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.760089 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:46.765553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.765524 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:58:46.814557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.814528 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e3670\"" Oct 30 21:58:46.814557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.814543 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e3670-dockercfg-8tkzn\"" Oct 30 21:58:46.832407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.832324 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:46.832563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.832406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fr5\" (UniqueName: \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:46.949289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.948739 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:46.949289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.948837 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fr5\" (UniqueName: \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:46.969024 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.950225 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:47.001434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.999520 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fr5\" (UniqueName: \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\") pod \"payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:47.001434 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:46.999593 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 21:58:47.018686 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.018659 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.034319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.027672 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3654d5\"" Oct 30 21:58:47.034319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.028521 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3654d5-dockercfg-n8k49\"" Oct 30 21:58:47.046624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.044559 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 21:58:47.080557 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.080526 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:47.133313 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.133245 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4"] Oct 30 21:58:47.206676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.200661 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51672719-7bdt4"] Oct 30 21:58:47.236291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.223290 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdkv7\" (UniqueName: \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.236291 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.223376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.325075 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.325028 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdkv7\" (UniqueName: \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.325284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.325129 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.326711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.326678 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.354066 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.354027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdkv7\" (UniqueName: \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\") pod \"payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.451302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.451239 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:47.546525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.545018 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:58:47.614907 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.609108 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" event={"ID":"087bcdc6-3341-4045-bc20-32683745f8e3","Type":"ContainerStarted","Data":"c93bcb72ad0b646971c91a63eec2d1632ac31b3f6bccdb2eeebc10535c7404ac"} Oct 30 21:58:47.777686 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:58:47.777655 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4940a5_b861_40d5_8c0e_6263d1cde106.slice/crio-e5f00b3439875a5a2fe9638430745c1edcbfe3e3800b92c2124e46824bd0fd2d WatchSource:0}: Error finding container e5f00b3439875a5a2fe9638430745c1edcbfe3e3800b92c2124e46824bd0fd2d: Status 404 returned error can't find the container with id e5f00b3439875a5a2fe9638430745c1edcbfe3e3800b92c2124e46824bd0fd2d Oct 30 21:58:47.777827 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.777801 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 21:58:47.779677 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:47.779661 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:58:48.047991 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.047347 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg"] Oct 30 21:58:48.059780 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.054806 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3ac509-bfcjg"] Oct 30 21:58:48.117233 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:58:48.117193 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:58:48.190942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.190906 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fec8940-34af-43ed-9bf9-b7c96c212e2c" path="/var/lib/kubelet/pods/4fec8940-34af-43ed-9bf9-b7c96c212e2c/volumes" Oct 30 21:58:48.191480 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.191457 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a13ed6-6da9-4a93-a463-8d1ea7506084" path="/var/lib/kubelet/pods/62a13ed6-6da9-4a93-a463-8d1ea7506084/volumes" Oct 30 21:58:48.191948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.191934 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969d71c0-f8cb-41e6-a677-4c2cd9bc63b0" path="/var/lib/kubelet/pods/969d71c0-f8cb-41e6-a677-4c2cd9bc63b0/volumes" Oct 30 21:58:48.192712 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.192696 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf740fe-410e-48e5-9ead-f25bdbfb2954" path="/var/lib/kubelet/pods/eaf740fe-410e-48e5-9ead-f25bdbfb2954/volumes" Oct 30 21:58:48.626862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.626821 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" event={"ID":"cc4940a5-b861-40d5-8c0e-6263d1cde106","Type":"ContainerStarted","Data":"e5f00b3439875a5a2fe9638430745c1edcbfe3e3800b92c2124e46824bd0fd2d"} Oct 30 21:58:48.630220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.630175 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" event={"ID":"087bcdc6-3341-4045-bc20-32683745f8e3","Type":"ContainerStarted","Data":"d643f4f4814c5106ba103b74133a00923363f6d47f20554e43a0ebdaa854186b"} Oct 30 21:58:48.630386 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.630313 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:48.657002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:48.656427 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" podStartSLOduration=2.656408528 podStartE2EDuration="2.656408528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:48.652472336 +0000 UTC m=+1152.602427302" watchObservedRunningTime="2026-10-30 21:58:48.656408528 +0000 UTC m=+1152.606363476" Oct 30 21:58:50.462062 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:50.461801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:58:50.467054 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:50.465502 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:58:50.656102 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:50.656058 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" event={"ID":"cc4940a5-b861-40d5-8c0e-6263d1cde106","Type":"ContainerStarted","Data":"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582"} Oct 30 21:58:50.656316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:50.656153 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:58:50.678617 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:50.678564 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" podStartSLOduration=2.132037617 podStartE2EDuration="4.67854614s" totalImagesPullingTime="2.546508523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:50.674023995 +0000 UTC m=+1154.623978970" watchObservedRunningTime="2026-10-30 21:58:50.67854614 +0000 UTC m=+1154.628501090" Oct 30 21:58:52.034294 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:52.028548 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm"] Oct 30 21:58:52.060317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:52.050477 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bf337-cv2xm"] Oct 30 21:58:52.204212 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:52.204173 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba85e63-e14f-4bac-ba05-f791ae093f37" path="/var/lib/kubelet/pods/5ba85e63-e14f-4bac-ba05-f791ae093f37/volumes" Oct 30 21:58:59.646668 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:59.646638 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:58:59.851650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:59.850905 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:58:59.851650 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:58:59.851210 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" containerID="cri-o://160634d683a6a5af06cf5f027bd72bda14eb4e6e4278bec7db3cf984b64c7881" gracePeriod=30 Oct 30 21:59:00.328863 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.328827 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 21:59:00.336350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.336216 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.340505 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.339601 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3654d5-openshift-default-dockercfg-h757h\"" Oct 30 21:59:00.343793 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.343772 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 21:59:00.361508 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361471 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.361654 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.361721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361689 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.361901 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.361970 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361916 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.361970 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.361951 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.362074 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.362051 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.362129 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.362093 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.362186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.362169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9cx\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464206 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464281 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464316 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464346 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464376 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464441 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9cx\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464573 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464624 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.464669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.465133 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.467289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.466113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.482288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.471455 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.482288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.479399 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.482288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.479641 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.535288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.531221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.541289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.537945 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.545290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.544590 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9cx\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.549286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.545775 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:00.667059 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:00.667026 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:01.002158 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:01.002129 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47de940a_79cf_418a_8983_35541566b94a.slice/crio-81d445bc2711bfb1292fd9f2d9e76b55b5ca9d7a9927c31d1dfc1168905ea179 WatchSource:0}: Error finding container 81d445bc2711bfb1292fd9f2d9e76b55b5ca9d7a9927c31d1dfc1168905ea179: Status 404 returned error can't find the container with id 81d445bc2711bfb1292fd9f2d9e76b55b5ca9d7a9927c31d1dfc1168905ea179 Oct 30 21:59:01.002873 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.002850 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 21:59:01.004861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.004821 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:01.005045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.004905 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:01.005045 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.004953 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:01.038692 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.038667 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k"] Oct 30 21:59:01.044398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.044350 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a715cd9e-9lc9k"] Oct 30 21:59:01.682295 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.679252 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 21:59:01.909949 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.909903 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" event={"ID":"47de940a-79cf-418a-8983-35541566b94a","Type":"ContainerStarted","Data":"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b"} Oct 30 21:59:01.910300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.909994 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" event={"ID":"47de940a-79cf-418a-8983-35541566b94a","Type":"ContainerStarted","Data":"81d445bc2711bfb1292fd9f2d9e76b55b5ca9d7a9927c31d1dfc1168905ea179"} Oct 30 21:59:01.959499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:01.952233 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podStartSLOduration=1.952209151 podStartE2EDuration="1.952209151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:01.942568518 +0000 UTC m=+1165.892523465" watchObservedRunningTime="2026-10-30 21:59:01.952209151 +0000 UTC m=+1165.902164100" Oct 30 21:59:02.003964 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.003924 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88"] Oct 30 21:59:02.008982 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.008960 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.022300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.014551 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:02.022300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.014668 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:02.022300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.015366 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:02.022300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.015399 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:59:02.077442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.059881 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88"] Oct 30 21:59:02.095692 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.095657 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67"] Oct 30 21:59:02.108596 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.108569 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a958dcf5-rbp67"] Oct 30 21:59:02.148358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.128023 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.148358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.128134 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.148358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.128221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drfn\" (UniqueName: \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.189751 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.189722 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c79123-31fd-4e2f-8e45-aed9939d834d" path="/var/lib/kubelet/pods/17c79123-31fd-4e2f-8e45-aed9939d834d/volumes" Oct 30 21:59:02.190199 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.190185 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14c2347-020d-4ae6-b2fd-8e00efcdb5dd" path="/var/lib/kubelet/pods/f14c2347-020d-4ae6-b2fd-8e00efcdb5dd/volumes" Oct 30 21:59:02.245684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.234284 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.245684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.234444 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drfn\" (UniqueName: \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.245684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.234812 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.245684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.239510 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.247944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.247919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.251707 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.251681 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drfn\" (UniqueName: \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\") pod \"maas-api-revoke-keys-e2e-api-a-004c17-wqb88\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.398861 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.398829 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:02.668401 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.668360 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:02.669417 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.669384 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:02.669556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.669463 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:02.763652 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:02.763323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88"] Oct 30 21:59:02.766742 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:02.766681 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e9a140_328d_498e_8a28_6743cf8745b9.slice/crio-b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1 WatchSource:0}: Error finding container b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1: Status 404 returned error can't find the container with id b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1 Oct 30 21:59:03.019669 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:03.019571 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" event={"ID":"35e9a140-328d-498e-8a28-6743cf8745b9","Type":"ContainerStarted","Data":"b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1"} Oct 30 21:59:03.677304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:03.671203 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:03.677304 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:03.671285 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:04.068944 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:04.068856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" event={"ID":"35e9a140-328d-498e-8a28-6743cf8745b9","Type":"ContainerStarted","Data":"e024d41f3a536f794380231ef0e73cf4048ebb47507b78597d8d23d606574921"} Oct 30 21:59:04.097076 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:04.097032 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" podStartSLOduration=3.097018364 podStartE2EDuration="3.097018364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:04.096205982 +0000 UTC m=+1168.046160928" watchObservedRunningTime="2026-10-30 21:59:04.097018364 +0000 UTC m=+1168.046973343" Oct 30 21:59:04.670022 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:04.669982 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:04.670350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:04.670317 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:05.087232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.087199 5036 generic.go:356] "Generic (PLEG): container finished" podID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerID="160634d683a6a5af06cf5f027bd72bda14eb4e6e4278bec7db3cf984b64c7881" exitCode=0 Oct 30 21:59:05.087810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.087297 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerDied","Data":"160634d683a6a5af06cf5f027bd72bda14eb4e6e4278bec7db3cf984b64c7881"} Oct 30 21:59:05.087810 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.087380 5036 scope.go:122] "RemoveContainer" containerID="432c3d6ee7ac53057ed9d918a7eb60908f4a86c46500b203e074554f572aa93f" Oct 30 21:59:05.090696 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.090674 5036 generic.go:356] "Generic (PLEG): container finished" podID="35e9a140-328d-498e-8a28-6743cf8745b9" containerID="e024d41f3a536f794380231ef0e73cf4048ebb47507b78597d8d23d606574921" exitCode=0 Oct 30 21:59:05.090854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.090753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" event={"ID":"35e9a140-328d-498e-8a28-6743cf8745b9","Type":"ContainerDied","Data":"e024d41f3a536f794380231ef0e73cf4048ebb47507b78597d8d23d606574921"} Oct 30 21:59:05.594546 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.591898 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:59:05.661195 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.661165 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.661371 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.661232 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.661566 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.661543 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:05.661743 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.661725 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.661948 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.661911 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:05.662595 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.662550 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:05.663388 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.663368 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.664030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.663995 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "kube-api-access-5c8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:05.664167 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.664128 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.664306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.664205 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.664306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.664260 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.664446 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.664530 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\") pod \"713fd7b8-b181-46be-b9f2-a49d40a31dda\" (UID: \"713fd7b8-b181-46be-b9f2-a49d40a31dda\") " Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665396 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665462 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c8rk\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-kube-api-access-5c8rk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665478 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665487 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.665560 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665495 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fd7b8-b181-46be-b9f2-a49d40a31dda-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.666067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:05.666067 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.665968 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:05.666958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.666936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:05.667742 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.667716 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:05.667832 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.667764 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:05.669198 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.669171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token" pod "713fd7b8-b181-46be-b9f2-a49d40a31dda" (UID: "713fd7b8-b181-46be-b9f2-a49d40a31dda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:05.767033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.767001 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.767033 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.767036 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.767231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.767047 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fd7b8-b181-46be-b9f2-a49d40a31dda-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.767231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.767056 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.767231 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:05.767066 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fd7b8-b181-46be-b9f2-a49d40a31dda-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:06.199078 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.198931 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" Oct 30 21:59:06.206558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.206527 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg" event={"ID":"713fd7b8-b181-46be-b9f2-a49d40a31dda","Type":"ContainerDied","Data":"416006c5825902163a190198f7495e64d28e14036851f9f838535f08b372aefc"} Oct 30 21:59:06.206705 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.206592 5036 scope.go:122] "RemoveContainer" containerID="160634d683a6a5af06cf5f027bd72bda14eb4e6e4278bec7db3cf984b64c7881" Oct 30 21:59:06.296814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.295354 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:59:06.300160 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.299502 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4bf337-openshift-default-5686bb5485-ktkbg"] Oct 30 21:59:06.490858 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.490825 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:06.585802 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.585771 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:06.586556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586540 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35e9a140-328d-498e-8a28-6743cf8745b9" containerName="revoke-keys" Oct 30 21:59:06.586606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586558 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e9a140-328d-498e-8a28-6743cf8745b9" containerName="revoke-keys" Oct 30 21:59:06.586606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586572 5036 cpu_manager.go:414] "containerMap: removing container" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.586606 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586578 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.586699 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586691 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e9a140-328d-498e-8a28-6743cf8745b9" containerName="revoke-keys" Oct 30 21:59:06.586732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586702 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.586732 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.586711 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.590565 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.590545 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:06.595544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.595525 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\") pod \"35e9a140-328d-498e-8a28-6743cf8745b9\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " Oct 30 21:59:06.595711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.595697 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\") pod \"35e9a140-328d-498e-8a28-6743cf8745b9\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " Oct 30 21:59:06.595847 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.595834 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\" (UniqueName: \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\") pod \"35e9a140-328d-498e-8a28-6743cf8745b9\" (UID: \"35e9a140-328d-498e-8a28-6743cf8745b9\") " Oct 30 21:59:06.596928 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.596903 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca" pod "35e9a140-328d-498e-8a28-6743cf8745b9" (UID: "35e9a140-328d-498e-8a28-6743cf8745b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:06.601995 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.600574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn" pod "35e9a140-328d-498e-8a28-6743cf8745b9" (UID: "35e9a140-328d-498e-8a28-6743cf8745b9"). InnerVolumeSpecName "kube-api-access-4drfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:06.602392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.601817 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c614f9\"" Oct 30 21:59:06.602392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.601983 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c614f9-dockercfg-ttscl\"" Oct 30 21:59:06.602392 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.601788 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca" pod "35e9a140-328d-498e-8a28-6743cf8745b9" (UID: "35e9a140-328d-498e-8a28-6743cf8745b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:06.608025 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.608005 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:06.659644 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.659616 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:06.667027 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.666999 5036 cpu_manager.go:414] "containerMap: removing container" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.667224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.667206 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" containerName="istio-proxy" Oct 30 21:59:06.668684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.668161 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:06.668684 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.668249 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:06.673350 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.673327 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:06.673457 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.673423 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:06.730359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.708990 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:06.730359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.709178 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:06.730359 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.709217 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5fm\" (UniqueName: \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:06.741725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.741679 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:06.840623 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.840589 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drfn\" (UniqueName: \"kubernetes.io/projected/35e9a140-328d-498e-8a28-6743cf8745b9-kube-api-access-4drfn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:06.840851 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.840830 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:06.840977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.840963 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e9a140-328d-498e-8a28-6743cf8745b9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:06.949361 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.949318 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:06.965563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.965526 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:06.966186 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.966159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5fm\" (UniqueName: \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:06.966850 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.966827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:06.995347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.950455 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:07.005292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:06.967772 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:07.018147 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.018116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\") pod \"payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:07.048664 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.048630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5fm\" (UniqueName: \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:07.209575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.209545 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" Oct 30 21:59:07.209575 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.209562 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88" event={"ID":"35e9a140-328d-498e-8a28-6743cf8745b9","Type":"ContainerDied","Data":"b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1"} Oct 30 21:59:07.210069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.209619 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0b1b0f1ce6bc697a3fb9bdf10348802dea0c962aaad76f56c95326fc12b81b1" Oct 30 21:59:07.259197 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.259115 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:07.301978 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.301943 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:07.599377 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.599349 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:07.668220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.667912 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:59:07.668220 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.667954 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 21:59:07.689352 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:07.689324 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:07.694476 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:07.694444 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ec5a1c_2ad7_48e8_9a98_b0c9b38a7b54.slice/crio-b3e92140bc2ab31581f408447354a2e31040247257a2d94c2b5399f860e8c3de WatchSource:0}: Error finding container b3e92140bc2ab31581f408447354a2e31040247257a2d94c2b5399f860e8c3de: Status 404 returned error can't find the container with id b3e92140bc2ab31581f408447354a2e31040247257a2d94c2b5399f860e8c3de Oct 30 21:59:08.194552 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.194462 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713fd7b8-b181-46be-b9f2-a49d40a31dda" path="/var/lib/kubelet/pods/713fd7b8-b181-46be-b9f2-a49d40a31dda/volumes" Oct 30 21:59:08.270204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.270151 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" event={"ID":"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54","Type":"ContainerStarted","Data":"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb"} Oct 30 21:59:08.270756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.270289 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" event={"ID":"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54","Type":"ContainerStarted","Data":"b3e92140bc2ab31581f408447354a2e31040247257a2d94c2b5399f860e8c3de"} Oct 30 21:59:08.271639 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.271476 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:08.280044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.278024 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" event={"ID":"198005fd-72d4-4db4-a3c7-a2c88460293b","Type":"ContainerStarted","Data":"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1"} Oct 30 21:59:08.280044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.278125 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" event={"ID":"198005fd-72d4-4db4-a3c7-a2c88460293b","Type":"ContainerStarted","Data":"985b0b8fa64888e2a2323da6fd3a2a0b5e33fd7a900083f4f3c674f5ac2a8b0d"} Oct 30 21:59:08.280044 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.278580 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:08.307523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.306988 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" podStartSLOduration=2.306974374 podStartE2EDuration="2.306974374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:08.30276506 +0000 UTC m=+1172.252720008" watchObservedRunningTime="2026-10-30 21:59:08.306974374 +0000 UTC m=+1172.256929319" Oct 30 21:59:08.672006 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.671978 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:08.672421 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.672391 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:08.673133 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.673117 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 21:59:08.708857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:08.708798 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" podStartSLOduration=2.708782875 podStartE2EDuration="2.708782875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:08.32849473 +0000 UTC m=+1172.278449676" watchObservedRunningTime="2026-10-30 21:59:08.708782875 +0000 UTC m=+1172.658737847" Oct 30 21:59:11.101916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.101885 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:59:11.901404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.901373 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 21:59:11.907809 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.907789 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:11.922734 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.922709 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0b3fe4\"" Oct 30 21:59:11.926688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.923234 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0b3fe4-dockercfg-q76jl\"" Oct 30 21:59:11.926688 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.925159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 21:59:11.951914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.951878 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 21:59:11.957558 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.957528 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:11.968834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:11.968802 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 21:59:12.090290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.085449 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.090290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.085496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8852\" (UniqueName: \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.090290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.085542 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782pq\" (UniqueName: \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.090290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.085780 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.212544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.212736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.212768 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8852\" (UniqueName: \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.212802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782pq\" (UniqueName: \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.214477 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.215513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.215186 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.262990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.259232 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782pq\" (UniqueName: \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.266331 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.263867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8852\" (UniqueName: \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\") pod \"payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.268124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.268096 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:59:12.280063 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.279725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:12.339500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.338236 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:59:12.539148 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.538853 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:12.699435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:12.699295 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 21:59:12.701876 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:12.701832 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b9c8e3_c4aa_4f1a_adea_677007e1ef8f.slice/crio-0be00325b3e0349b29a991765e95aafb713b8bb0de612b2563fa4453d5ceb3ee WatchSource:0}: Error finding container 0be00325b3e0349b29a991765e95aafb713b8bb0de612b2563fa4453d5ceb3ee: Status 404 returned error can't find the container with id 0be00325b3e0349b29a991765e95aafb713b8bb0de612b2563fa4453d5ceb3ee Oct 30 21:59:13.204934 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.204900 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 21:59:13.837288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.837190 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" event={"ID":"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f","Type":"ContainerStarted","Data":"a5b5348816f496d591ad2326ccd2a49954976d1339cada3f19fd8c26eae47fd1"} Oct 30 21:59:13.837288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.837254 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" event={"ID":"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f","Type":"ContainerStarted","Data":"0be00325b3e0349b29a991765e95aafb713b8bb0de612b2563fa4453d5ceb3ee"} Oct 30 21:59:13.837872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.837304 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:13.892340 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.840076 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" event={"ID":"e894d868-4d71-4662-8559-02f26690c377","Type":"ContainerStarted","Data":"9cc35947917bcb1afc476e8fcca04185cddb956f133fad404b7b7a8bf79ddc7d"} Oct 30 21:59:13.962689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:13.953686 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" podStartSLOduration=2.953662724 podStartE2EDuration="2.953662724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:13.928720222 +0000 UTC m=+1177.878675169" watchObservedRunningTime="2026-10-30 21:59:13.953662724 +0000 UTC m=+1177.903617671" Oct 30 21:59:14.395825 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:14.395792 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg"] Oct 30 21:59:14.395977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:14.395841 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ed0a9e-zbjfg"] Oct 30 21:59:15.006715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:15.006602 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" event={"ID":"e894d868-4d71-4662-8559-02f26690c377","Type":"ContainerStarted","Data":"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857"} Oct 30 21:59:15.006715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:15.006704 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:15.108335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:15.106157 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" podStartSLOduration=4.106143166 podStartE2EDuration="4.106143166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:15.078734553 +0000 UTC m=+1179.028689512" watchObservedRunningTime="2026-10-30 21:59:15.106143166 +0000 UTC m=+1179.056098156" Oct 30 21:59:16.102986 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:16.102947 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" podUID="087bcdc6-3341-4045-bc20-32683745f8e3" containerName="payload-processing" containerID="cri-o://d643f4f4814c5106ba103b74133a00923363f6d47f20554e43a0ebdaa854186b" gracePeriod=25 Oct 30 21:59:16.204744 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:16.204703 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b04ad9d-bb33-4039-a085-ca8ec7d0c320" path="/var/lib/kubelet/pods/5b04ad9d-bb33-4039-a085-ca8ec7d0c320/volumes" Oct 30 21:59:17.279482 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.277339 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" podUID="ccd4e3ec-5f03-4611-aba9-b394d024359d" containerName="payload-processing" containerID="cri-o://fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426" gracePeriod=25 Oct 30 21:59:17.301523 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.301489 5036 generic.go:356] "Generic (PLEG): container finished" podID="087bcdc6-3341-4045-bc20-32683745f8e3" containerID="d643f4f4814c5106ba103b74133a00923363f6d47f20554e43a0ebdaa854186b" exitCode=0 Oct 30 21:59:17.301953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.301933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" event={"ID":"087bcdc6-3341-4045-bc20-32683745f8e3","Type":"ContainerDied","Data":"d643f4f4814c5106ba103b74133a00923363f6d47f20554e43a0ebdaa854186b"} Oct 30 21:59:17.343286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.343117 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" podUID="f0ea4e15-20b4-495b-926c-5c9348176e44" containerName="payload-pre-processing" containerID="cri-o://faf94fc673715a1bc75ce9b7f6fd20def0f9c0bc6f2378ccb8aa2dc04c1c6b59" gracePeriod=25 Oct 30 21:59:17.707844 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.707814 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:59:17.798933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.797737 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\" (UniqueName: \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\") pod \"087bcdc6-3341-4045-bc20-32683745f8e3\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " Oct 30 21:59:17.798933 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.797933 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\") pod \"087bcdc6-3341-4045-bc20-32683745f8e3\" (UID: \"087bcdc6-3341-4045-bc20-32683745f8e3\") " Oct 30 21:59:17.800756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.800732 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume" pod "087bcdc6-3341-4045-bc20-32683745f8e3" (UID: "087bcdc6-3341-4045-bc20-32683745f8e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:17.807316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.805934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5" pod "087bcdc6-3341-4045-bc20-32683745f8e3" (UID: "087bcdc6-3341-4045-bc20-32683745f8e3"). InnerVolumeSpecName "kube-api-access-57fr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:17.943293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.902645 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087bcdc6-3341-4045-bc20-32683745f8e3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.943293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:17.902687 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fr5\" (UniqueName: \"kubernetes.io/projected/087bcdc6-3341-4045-bc20-32683745f8e3-kube-api-access-57fr5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.152293 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:59:18.137166 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:59:18.209053 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.209017 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:59:18.387088 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.387043 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\") pod \"ccd4e3ec-5f03-4611-aba9-b394d024359d\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " Oct 30 21:59:18.387647 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.387351 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\" (UniqueName: \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\") pod \"ccd4e3ec-5f03-4611-aba9-b394d024359d\" (UID: \"ccd4e3ec-5f03-4611-aba9-b394d024359d\") " Oct 30 21:59:18.388230 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.388102 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume" pod "ccd4e3ec-5f03-4611-aba9-b394d024359d" (UID: "ccd4e3ec-5f03-4611-aba9-b394d024359d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.395903 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.395853 5036 generic.go:356] "Generic (PLEG): container finished" podID="f0ea4e15-20b4-495b-926c-5c9348176e44" containerID="faf94fc673715a1bc75ce9b7f6fd20def0f9c0bc6f2378ccb8aa2dc04c1c6b59" exitCode=0 Oct 30 21:59:18.396479 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.395972 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" event={"ID":"f0ea4e15-20b4-495b-926c-5c9348176e44","Type":"ContainerDied","Data":"faf94fc673715a1bc75ce9b7f6fd20def0f9c0bc6f2378ccb8aa2dc04c1c6b59"} Oct 30 21:59:18.403470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.402321 5036 generic.go:356] "Generic (PLEG): container finished" podID="ccd4e3ec-5f03-4611-aba9-b394d024359d" containerID="fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426" exitCode=0 Oct 30 21:59:18.403470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.402458 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" Oct 30 21:59:18.403470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.402620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" event={"ID":"ccd4e3ec-5f03-4611-aba9-b394d024359d","Type":"ContainerDied","Data":"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426"} Oct 30 21:59:18.403470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.402678 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s" event={"ID":"ccd4e3ec-5f03-4611-aba9-b394d024359d","Type":"ContainerDied","Data":"33ca2ee216aed3f467efcc9d6f5f100782bfc116375cb0c0b5433e476f5605c7"} Oct 30 21:59:18.403470 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.402708 5036 scope.go:122] "RemoveContainer" containerID="fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426" Oct 30 21:59:18.414288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.410700 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh" pod "ccd4e3ec-5f03-4611-aba9-b394d024359d" (UID: "ccd4e3ec-5f03-4611-aba9-b394d024359d"). InnerVolumeSpecName "kube-api-access-m5xgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.493939 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.493905 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" event={"ID":"087bcdc6-3341-4045-bc20-32683745f8e3","Type":"ContainerDied","Data":"c93bcb72ad0b646971c91a63eec2d1632ac31b3f6bccdb2eeebc10535c7404ac"} Oct 30 21:59:18.494237 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.494207 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9" Oct 30 21:59:18.496214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.494049 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xgh\" (UniqueName: \"kubernetes.io/projected/ccd4e3ec-5f03-4611-aba9-b394d024359d-kube-api-access-m5xgh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.496398 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.496379 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4e3ec-5f03-4611-aba9-b394d024359d-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.649838 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.607529 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:59:18.692663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.651644 5036 scope.go:122] "RemoveContainer" containerID="fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426" Oct 30 21:59:18.692663 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:59:18.654747 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426\": container with ID starting with fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426 not found: ID does not exist" containerID="fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426" Oct 30 21:59:18.692663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.654801 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426"} err="failed to get container status \"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426\": rpc error: code = NotFound desc = could not find container \"fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426\": container with ID starting with fba7a1d7c279695b8b2738cda204293ecba9d1dff5a064fc25555d48fe452426 not found: ID does not exist" Oct 30 21:59:18.692663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.654831 5036 scope.go:122] "RemoveContainer" containerID="d643f4f4814c5106ba103b74133a00923363f6d47f20554e43a0ebdaa854186b" Oct 30 21:59:18.748305 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.747543 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:59:18.769339 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.768534 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-689f6f6d75-scpn9"] Oct 30 21:59:18.777693 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.777660 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:59:18.794154 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.791355 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-004c17-55b855d645-klk2s"] Oct 30 21:59:18.910002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.908599 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\" (UniqueName: \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\") pod \"f0ea4e15-20b4-495b-926c-5c9348176e44\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " Oct 30 21:59:18.910002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.908657 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\") pod \"f0ea4e15-20b4-495b-926c-5c9348176e44\" (UID: \"f0ea4e15-20b4-495b-926c-5c9348176e44\") " Oct 30 21:59:18.910002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.909520 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume" pod "f0ea4e15-20b4-495b-926c-5c9348176e44" (UID: "f0ea4e15-20b4-495b-926c-5c9348176e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.919507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.910173 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ea4e15-20b4-495b-926c-5c9348176e44-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.954150 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:18.954072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h" pod "f0ea4e15-20b4-495b-926c-5c9348176e44" (UID: "f0ea4e15-20b4-495b-926c-5c9348176e44"). InnerVolumeSpecName "kube-api-access-2n68h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:19.024093 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.011446 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n68h\" (UniqueName: \"kubernetes.io/projected/f0ea4e15-20b4-495b-926c-5c9348176e44-kube-api-access-2n68h\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:19.076594 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.076556 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078398 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f0ea4e15-20b4-495b-926c-5c9348176e44" containerName="payload-pre-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078424 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ea4e15-20b4-495b-926c-5c9348176e44" containerName="payload-pre-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078506 5036 cpu_manager.go:414] "containerMap: removing container" podUID="087bcdc6-3341-4045-bc20-32683745f8e3" containerName="payload-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078515 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087bcdc6-3341-4045-bc20-32683745f8e3" containerName="payload-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078541 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4e3ec-5f03-4611-aba9-b394d024359d" containerName="payload-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078548 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4e3ec-5f03-4611-aba9-b394d024359d" containerName="payload-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078710 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4e3ec-5f03-4611-aba9-b394d024359d" containerName="payload-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078723 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ea4e15-20b4-495b-926c-5c9348176e44" containerName="payload-pre-processing" Oct 30 21:59:19.078814 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.078744 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="087bcdc6-3341-4045-bc20-32683745f8e3" containerName="payload-processing" Oct 30 21:59:19.085011 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.084989 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.102942 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.100554 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 21:59:19.142162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.141540 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8c6cf1\"" Oct 30 21:59:19.142162 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.141700 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8c6cf1-dockercfg-swl7d\"" Oct 30 21:59:19.247433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.239013 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpr2\" (UniqueName: \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.247433 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.239201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.253069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.253038 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 21:59:19.262498 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.262448 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 21:59:19.262629 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.262597 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.268598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.267334 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e3670\"" Oct 30 21:59:19.277556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.277457 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 21:59:19.277556 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.277505 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 21:59:19.277737 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.277572 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.284473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.283556 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e3670-dockercfg-klzt6\"" Oct 30 21:59:19.293151 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.293132 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:19.359446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.359387 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpr2\" (UniqueName: \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.359840 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.359808 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.359989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.359949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjdq\" (UniqueName: \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.360051 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.359997 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.360562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.360237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.360562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.360502 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt27\" (UniqueName: \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.382316 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.378246 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.510347 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.475895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjdq\" (UniqueName: \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.617299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.609621 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.617299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.609780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.617299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.609915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt27\" (UniqueName: \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.617299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.617007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.625288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.620032 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.688300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.688038 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpr2\" (UniqueName: \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\") pod \"payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.694779 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.694708 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjdq\" (UniqueName: \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\") pod \"payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:19.697191 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.697153 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt27\" (UniqueName: \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.769492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.769128 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:19.787328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.785665 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" event={"ID":"f0ea4e15-20b4-495b-926c-5c9348176e44","Type":"ContainerDied","Data":"3bd17530ad710398f759250c9bf977f7b28b650810b8881a202675b165084719"} Oct 30 21:59:19.787328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.785745 5036 scope.go:122] "RemoveContainer" containerID="faf94fc673715a1bc75ce9b7f6fd20def0f9c0bc6f2378ccb8aa2dc04c1c6b59" Oct 30 21:59:19.787328 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.785875 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v" Oct 30 21:59:19.863517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.863452 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:59:19.885568 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.885493 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-004c17-8557cc59c5-ms56v"] Oct 30 21:59:19.888530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.888151 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:19.925914 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:19.922899 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:20.197472 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.197441 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087bcdc6-3341-4045-bc20-32683745f8e3" path="/var/lib/kubelet/pods/087bcdc6-3341-4045-bc20-32683745f8e3/volumes" Oct 30 21:59:20.198441 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.198414 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4e3ec-5f03-4611-aba9-b394d024359d" path="/var/lib/kubelet/pods/ccd4e3ec-5f03-4611-aba9-b394d024359d/volumes" Oct 30 21:59:20.199307 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.199284 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ea4e15-20b4-495b-926c-5c9348176e44" path="/var/lib/kubelet/pods/f0ea4e15-20b4-495b-926c-5c9348176e44/volumes" Oct 30 21:59:20.237242 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:20.237209 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb27bc44_3464_47a1_8426_e770ffe15eda.slice/crio-fd4425f435f68e02b6842d9b0afae3cf242682507d39e4b021be4d472c2e58a1 WatchSource:0}: Error finding container fd4425f435f68e02b6842d9b0afae3cf242682507d39e4b021be4d472c2e58a1: Status 404 returned error can't find the container with id fd4425f435f68e02b6842d9b0afae3cf242682507d39e4b021be4d472c2e58a1 Oct 30 21:59:20.239142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.239119 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 21:59:20.316967 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:20.316935 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fb2247_fb37_4e64_81df_47ffaa1c38b5.slice/crio-33a3c2e2fc608a0159bdcfb03aa2d95414a9a30d89cb3922f1bae777eba4dbe4 WatchSource:0}: Error finding container 33a3c2e2fc608a0159bdcfb03aa2d95414a9a30d89cb3922f1bae777eba4dbe4: Status 404 returned error can't find the container with id 33a3c2e2fc608a0159bdcfb03aa2d95414a9a30d89cb3922f1bae777eba4dbe4 Oct 30 21:59:20.321676 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.321649 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 21:59:20.331306 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:20.330816 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b962ac_f06e_4e57_92a5_74248dcd4eb1.slice/crio-fe013f24c245d33c7b482d749717946c39feb1c002eb5cbd4de3a4fb980f699c WatchSource:0}: Error finding container fe013f24c245d33c7b482d749717946c39feb1c002eb5cbd4de3a4fb980f699c: Status 404 returned error can't find the container with id fe013f24c245d33c7b482d749717946c39feb1c002eb5cbd4de3a4fb980f699c Oct 30 21:59:20.331776 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.331759 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 21:59:20.341404 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.341384 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:20.822927 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.822892 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" event={"ID":"23fb2247-fb37-4e64-81df-47ffaa1c38b5","Type":"ContainerStarted","Data":"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311"} Oct 30 21:59:20.823553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.822953 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" event={"ID":"23fb2247-fb37-4e64-81df-47ffaa1c38b5","Type":"ContainerStarted","Data":"33a3c2e2fc608a0159bdcfb03aa2d95414a9a30d89cb3922f1bae777eba4dbe4"} Oct 30 21:59:20.824692 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.824662 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:20.829001 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.828966 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" event={"ID":"db27bc44-3464-47a1-8426-e770ffe15eda","Type":"ContainerStarted","Data":"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416"} Oct 30 21:59:20.829143 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.829031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" event={"ID":"db27bc44-3464-47a1-8426-e770ffe15eda","Type":"ContainerStarted","Data":"fd4425f435f68e02b6842d9b0afae3cf242682507d39e4b021be4d472c2e58a1"} Oct 30 21:59:20.829633 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.829340 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:20.831811 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.831772 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" event={"ID":"52b962ac-f06e-4e57-92a5-74248dcd4eb1","Type":"ContainerStarted","Data":"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715"} Oct 30 21:59:20.831916 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.831840 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" event={"ID":"52b962ac-f06e-4e57-92a5-74248dcd4eb1","Type":"ContainerStarted","Data":"fe013f24c245d33c7b482d749717946c39feb1c002eb5cbd4de3a4fb980f699c"} Oct 30 21:59:20.833008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.832979 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:20.850871 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.850811 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" podStartSLOduration=1.850791726 podStartE2EDuration="1.850791726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:20.84678637 +0000 UTC m=+1184.796741316" watchObservedRunningTime="2026-10-30 21:59:20.850791726 +0000 UTC m=+1184.800746682" Oct 30 21:59:20.867119 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.867056 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" podStartSLOduration=1.867040102 podStartE2EDuration="1.867040102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:20.866016327 +0000 UTC m=+1184.815971275" watchObservedRunningTime="2026-10-30 21:59:20.867040102 +0000 UTC m=+1184.816995049" Oct 30 21:59:20.886138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:20.886079 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" podStartSLOduration=1.886061188 podStartE2EDuration="1.886061188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:20.884366373 +0000 UTC m=+1184.834321321" watchObservedRunningTime="2026-10-30 21:59:20.886061188 +0000 UTC m=+1184.836016135" Oct 30 21:59:21.054450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:21.054322 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5"] Oct 30 21:59:21.093430 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:21.080874 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ee3639-bbff5"] Oct 30 21:59:22.192822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:22.192771 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e" path="/var/lib/kubelet/pods/c3139f67-5130-4a7a-b5d5-e2dd2ea36e5e/volumes" Oct 30 21:59:23.366491 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:23.366447 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:23.366979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:23.366504 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:24.694473 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:24.694425 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:25.402366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:24.694580 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:25.402366 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.009868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 21:59:25.921363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.921325 5036 generic.go:356] "Generic (PLEG): container finished" podID="a316ac0c-a489-466e-a974-40530fa15cae" containerID="de8dc85eb7fc4ceaaeeb97e10b10c1f4a98ed796122ee5278d6be3acca77d408" exitCode=0 Oct 30 21:59:25.921818 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.921409 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerDied","Data":"de8dc85eb7fc4ceaaeeb97e10b10c1f4a98ed796122ee5278d6be3acca77d408"} Oct 30 21:59:25.921978 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.921954 5036 scope.go:122] "RemoveContainer" containerID="de8dc85eb7fc4ceaaeeb97e10b10c1f4a98ed796122ee5278d6be3acca77d408" Oct 30 21:59:25.923812 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.923771 5036 generic.go:356] "Generic (PLEG): container finished" podID="6dbb1593-f067-4962-af30-67e3758a305d" containerID="c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f" exitCode=0 Oct 30 21:59:25.923935 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.923873 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerDied","Data":"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f"} Oct 30 21:59:25.924242 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.924205 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:25.924335 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.924320 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:25.924373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.924333 5036 scope.go:122] "RemoveContainer" containerID="c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f" Oct 30 21:59:25.924407 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.924368 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:25.926015 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.925980 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:25.926146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.926068 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:25.926146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:25.926126 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:26.054302 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:26.053070 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 21:59:26.942325 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:26.942235 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerStarted","Data":"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776"} Oct 30 21:59:26.946200 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:26.946164 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerStarted","Data":"3096fe16f82ed942b7e90e977d83018e2733621e9934218600693864bb1dda09"} Oct 30 21:59:27.378701 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.378598 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:27.378908 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.378863 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.378976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.378935 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:27.718020 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.717566 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:59:27.718020 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.717699 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.718020 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:27.717755 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:28.481288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.480989 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.481288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.481046 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:28.778357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.747451 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.778357 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.747515 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:28.955293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.932091 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88"] Oct 30 21:59:28.981784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:28.981746 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-004c17-wqb88"] Oct 30 21:59:29.374296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:29.373719 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.374296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:29.373826 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:29.733290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:29.732681 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.733290 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:29.732764 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:30.201224 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.201189 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e9a140-328d-498e-8a28-6743cf8745b9" path="/var/lib/kubelet/pods/35e9a140-328d-498e-8a28-6743cf8745b9/volumes" Oct 30 21:59:30.374706 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.374668 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:30.375046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.375015 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:30.503921 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.503845 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:30.586836 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.585400 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:30.736296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.720660 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:30.736296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:30.720719 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:31.101428 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.101395 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp"] Oct 30 21:59:31.154791 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.154753 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e45bae-2pcqp"] Oct 30 21:59:31.381355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.374675 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.381355 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.374737 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:31.738358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.732657 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.738358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.732713 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:31.874285 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:31.874169 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 21:59:32.103156 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.103070 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5"] Oct 30 21:59:32.166333 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.136007 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140b1cb9-6lcj5"] Oct 30 21:59:32.195906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.195873 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5" path="/var/lib/kubelet/pods/1e8c8f14-3310-4f6a-9e55-5c010b8fd8d5/volumes" Oct 30 21:59:32.196391 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.196376 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3efb4f7-a766-4a73-af33-2fc5903ddc55" path="/var/lib/kubelet/pods/f3efb4f7-a766-4a73-af33-2fc5903ddc55/volumes" Oct 30 21:59:32.196878 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.196848 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp"] Oct 30 21:59:32.196989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.196885 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50257ce-ht6jp"] Oct 30 21:59:32.373813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.373709 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.373813 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.373794 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:32.733289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.728657 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.733289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.728720 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:32.871288 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.870986 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 21:59:32.884303 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:32.884253 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 21:59:33.391309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:33.383786 5036 patch_prober.go:29] interesting pod/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:59:33.391309 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:33.383846 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:59:33.715419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:33.715323 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:59:33.715419 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:33.715406 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 21:59:34.256487 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.256350 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef49a238-073d-4ce2-9f7c-3b1391196bde" path="/var/lib/kubelet/pods/ef49a238-073d-4ce2-9f7c-3b1391196bde/volumes" Oct 30 21:59:34.669046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.669016 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:34.669584 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.669565 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:34.670517 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.670487 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:34.720950 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.720911 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:59:34.721525 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.721502 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:59:34.722259 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:34.722241 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 21:59:35.504862 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:35.504823 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" podUID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" containerName="payload-processing" containerID="cri-o://0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb" gracePeriod=25 Oct 30 21:59:35.586002 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:35.585946 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" podUID="198005fd-72d4-4db4-a3c7-a2c88460293b" containerName="payload-pre-processing" containerID="cri-o://c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1" gracePeriod=25 Oct 30 21:59:35.907964 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:35.907593 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:35.958174 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:35.958144 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:36.044317 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.044208 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\") pod \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " Oct 30 21:59:36.044500 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.044464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\" (UniqueName: \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\") pod \"198005fd-72d4-4db4-a3c7-a2c88460293b\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " Oct 30 21:59:36.044570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.044496 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\") pod \"198005fd-72d4-4db4-a3c7-a2c88460293b\" (UID: \"198005fd-72d4-4db4-a3c7-a2c88460293b\") " Oct 30 21:59:36.044570 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.044548 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\") pod \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\" (UID: \"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54\") " Oct 30 21:59:36.045896 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.045864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume" pod "25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" (UID: "25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:36.046284 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.046242 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume" pod "198005fd-72d4-4db4-a3c7-a2c88460293b" (UID: "198005fd-72d4-4db4-a3c7-a2c88460293b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:36.047319 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.047258 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw" pod "25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" (UID: "25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54"). InnerVolumeSpecName "kube-api-access-4wbhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:36.047541 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.047522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm" pod "198005fd-72d4-4db4-a3c7-a2c88460293b" (UID: "198005fd-72d4-4db4-a3c7-a2c88460293b"). InnerVolumeSpecName "kube-api-access-7c5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:36.158805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.156887 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5fm\" (UniqueName: \"kubernetes.io/projected/198005fd-72d4-4db4-a3c7-a2c88460293b-kube-api-access-7c5fm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.158805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.156927 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198005fd-72d4-4db4-a3c7-a2c88460293b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.158805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.156943 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.158805 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.156956 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54-kube-api-access-4wbhw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.490926 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.490898 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:59:36.520838 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.520809 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 21:59:36.567107 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.567071 5036 generic.go:356] "Generic (PLEG): container finished" podID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" containerID="0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb" exitCode=0 Oct 30 21:59:36.567337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.567130 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" Oct 30 21:59:36.567337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.567152 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" event={"ID":"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54","Type":"ContainerDied","Data":"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb"} Oct 30 21:59:36.567337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.567246 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz" event={"ID":"25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54","Type":"ContainerDied","Data":"b3e92140bc2ab31581f408447354a2e31040247257a2d94c2b5399f860e8c3de"} Oct 30 21:59:36.567337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.567297 5036 scope.go:122] "RemoveContainer" containerID="0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb" Oct 30 21:59:36.569256 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.569228 5036 generic.go:356] "Generic (PLEG): container finished" podID="198005fd-72d4-4db4-a3c7-a2c88460293b" containerID="c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1" exitCode=0 Oct 30 21:59:36.569549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.569531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" Oct 30 21:59:36.569749 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.569715 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" event={"ID":"198005fd-72d4-4db4-a3c7-a2c88460293b","Type":"ContainerDied","Data":"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1"} Oct 30 21:59:36.569834 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.569777 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt" event={"ID":"198005fd-72d4-4db4-a3c7-a2c88460293b","Type":"ContainerDied","Data":"985b0b8fa64888e2a2323da6fd3a2a0b5e33fd7a900083f4f3c674f5ac2a8b0d"} Oct 30 21:59:36.580400 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.580384 5036 scope.go:122] "RemoveContainer" containerID="0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb" Oct 30 21:59:36.580657 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:59:36.580637 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb\": container with ID starting with 0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb not found: ID does not exist" containerID="0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb" Oct 30 21:59:36.580725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.580664 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb"} err="failed to get container status \"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb\": rpc error: code = NotFound desc = could not find container \"0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb\": container with ID starting with 0f63a0f3a510e862fe99adf966dddcd07404d980181649bbe672f4aec86a29eb not found: ID does not exist" Oct 30 21:59:36.580725 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.580679 5036 scope.go:122] "RemoveContainer" containerID="c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1" Oct 30 21:59:36.591854 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.591836 5036 scope.go:122] "RemoveContainer" containerID="c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1" Oct 30 21:59:36.592141 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:59:36.592117 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1\": container with ID starting with c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1 not found: ID does not exist" containerID="c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1" Oct 30 21:59:36.592240 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.592152 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1"} err="failed to get container status \"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1\": rpc error: code = NotFound desc = could not find container \"c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1\": container with ID starting with c555a8e09fca0dcd2e6f935625de6745ddc592e1f612da5894a0b8af90cd6ab1 not found: ID does not exist" Oct 30 21:59:36.594956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.594938 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:36.599504 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.599486 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-5b8f84fd7f-v2bwz"] Oct 30 21:59:36.621511 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.621492 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:36.628122 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.628101 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-79fb6474f9-6pbpt"] Oct 30 21:59:36.887748 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.887714 5036 scope.go:122] "RemoveContainer" containerID="28a1710b51ab07dc5e803171c820e4f8831ce4ebd380912b2992ce6f9c9a8ba0" Oct 30 21:59:36.899721 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.899696 5036 scope.go:122] "RemoveContainer" containerID="94bad482edcb626971532a1eafb95c24d3589ef863c315a15f020b2e04c47880" Oct 30 21:59:36.910910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.910889 5036 scope.go:122] "RemoveContainer" containerID="1c29f033b9cba01a55a2e746b6b48bafbd6b83ed8f682aedcd54c51cc8bd7ecd" Oct 30 21:59:36.912046 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912029 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 21:59:36.912822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912805 5036 cpu_manager.go:414] "containerMap: removing container" podUID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" containerName="payload-processing" Oct 30 21:59:36.912822 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912821 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" containerName="payload-processing" Oct 30 21:59:36.912979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912867 5036 cpu_manager.go:414] "containerMap: removing container" podUID="198005fd-72d4-4db4-a3c7-a2c88460293b" containerName="payload-pre-processing" Oct 30 21:59:36.912979 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912873 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198005fd-72d4-4db4-a3c7-a2c88460293b" containerName="payload-pre-processing" Oct 30 21:59:36.913082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.912985 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" containerName="payload-processing" Oct 30 21:59:36.913082 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.913000 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="198005fd-72d4-4db4-a3c7-a2c88460293b" containerName="payload-pre-processing" Oct 30 21:59:36.918619 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.918597 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:36.924375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.923519 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-19cab1\"" Oct 30 21:59:36.924375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.923642 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-19cab1-dockercfg-jskqf\"" Oct 30 21:59:36.924375 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.923662 5036 scope.go:122] "RemoveContainer" containerID="acb4d691042a58fd97ef1c924d040d8e91d1a2084ceb1899acda7eeecd93fa63" Oct 30 21:59:36.935349 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.935329 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 21:59:36.947553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.947530 5036 scope.go:122] "RemoveContainer" containerID="dc297f16ce28f294703398cba68800d0746d2fcd985ef7e3c82c72e4d508615a" Oct 30 21:59:36.963118 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.962121 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 21:59:36.971562 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.970505 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:36.975450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.975427 5036 scope.go:122] "RemoveContainer" containerID="437d53e358f79812c551cab8445a59c9597b600959c2ed087d10a426bdd61521" Oct 30 21:59:36.977300 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:36.977277 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 21:59:37.067690 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.067660 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:59:37.076084 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.076053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdj2x\" (UniqueName: \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.076318 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.076299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.076383 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.076344 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.076446 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.076389 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmzbp\" (UniqueName: \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.187411 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.187369 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.196387 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.196353 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.196652 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.196633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmzbp\" (UniqueName: \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.196920 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.196903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdj2x\" (UniqueName: \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.209326 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.193673 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.221299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.210447 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.288054 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.273035 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdj2x\" (UniqueName: \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\") pod \"payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.288054 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.287989 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmzbp\" (UniqueName: \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.324777 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.324738 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:37.561190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.561157 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:37.578544 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.578354 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5"] Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.588221 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5"] Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.588358 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.591330 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" containerID="cri-o://3096fe16f82ed942b7e90e977d83018e2733621e9934218600693864bb1dda09" gracePeriod=30 Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.592053 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.592126 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.592199 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:59:37.593553 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.592951 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:37.616975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.616086 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.616975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.616147 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.616975 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.616514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k965x\" (UniqueName: \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.722515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.720827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k965x\" (UniqueName: \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.722515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.720985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.722515 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.721033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.722946 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.722652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.725865 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.723848 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 21:59:37.730930 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.730904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.731387 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:37.731360 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8e343e_dbc7_4866_beec_2651bf37a636.slice/crio-82aae333c8162f6e718fba308a92dce81723fb5df4d3cb14bb4e311c94e646b6 WatchSource:0}: Error finding container 82aae333c8162f6e718fba308a92dce81723fb5df4d3cb14bb4e311c94e646b6: Status 404 returned error can't find the container with id 82aae333c8162f6e718fba308a92dce81723fb5df4d3cb14bb4e311c94e646b6 Oct 30 21:59:37.735112 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.734012 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k965x\" (UniqueName: \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\") pod \"maas-api-revoke-keys-e2e-api-b-f47f79-59tk5\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:37.969180 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:37.969152 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c4ccea_7e6b_417f_bbf5_fb4caeebdca4.slice/crio-2436625463a583b81b36d03a2d280dbfdd40e87d3bc603c94579461aeb5149d4 WatchSource:0}: Error finding container 2436625463a583b81b36d03a2d280dbfdd40e87d3bc603c94579461aeb5149d4: Status 404 returned error can't find the container with id 2436625463a583b81b36d03a2d280dbfdd40e87d3bc603c94579461aeb5149d4 Oct 30 21:59:37.969990 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.969966 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 21:59:37.971689 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:37.971669 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:38.192145 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.192109 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198005fd-72d4-4db4-a3c7-a2c88460293b" path="/var/lib/kubelet/pods/198005fd-72d4-4db4-a3c7-a2c88460293b/volumes" Oct 30 21:59:38.192742 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.192720 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54" path="/var/lib/kubelet/pods/25ec5a1c-2ad7-48e8-9a98-b0c9b38a7b54/volumes" Oct 30 21:59:38.271821 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.271793 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5"] Oct 30 21:59:38.272378 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:38.272355 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3836a17c_9eeb_4bd0_bd17_8f77fc66e340.slice/crio-af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee WatchSource:0}: Error finding container af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee: Status 404 returned error can't find the container with id af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee Oct 30 21:59:38.658296 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.656303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" event={"ID":"3836a17c-9eeb-4bd0-bd17-8f77fc66e340","Type":"ContainerStarted","Data":"af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee"} Oct 30 21:59:38.659395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.659154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" event={"ID":"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4","Type":"ContainerStarted","Data":"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1"} Oct 30 21:59:38.659395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.659204 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" event={"ID":"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4","Type":"ContainerStarted","Data":"2436625463a583b81b36d03a2d280dbfdd40e87d3bc603c94579461aeb5149d4"} Oct 30 21:59:38.659395 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.659250 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:38.662358 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.662330 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" event={"ID":"4d8e343e-dbc7-4866-beec-2651bf37a636","Type":"ContainerStarted","Data":"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0"} Oct 30 21:59:38.662530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.662394 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" event={"ID":"4d8e343e-dbc7-4866-beec-2651bf37a636","Type":"ContainerStarted","Data":"82aae333c8162f6e718fba308a92dce81723fb5df4d3cb14bb4e311c94e646b6"} Oct 30 21:59:38.662530 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.662424 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:38.681717 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.681670 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" podStartSLOduration=2.681658242 podStartE2EDuration="2.681658242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:38.679684851 +0000 UTC m=+1202.629639796" watchObservedRunningTime="2026-10-30 21:59:38.681658242 +0000 UTC m=+1202.631613208" Oct 30 21:59:38.701204 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:38.701140 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" podStartSLOduration=2.701118042 podStartE2EDuration="2.701118042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:38.698037927 +0000 UTC m=+1202.647992872" watchObservedRunningTime="2026-10-30 21:59:38.701118042 +0000 UTC m=+1202.651072989" Oct 30 21:59:39.683255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.683159 5036 generic.go:356] "Generic (PLEG): container finished" podID="3836a17c-9eeb-4bd0-bd17-8f77fc66e340" containerID="adb2383ef084693422561370d2c21bc5f891dfa510f33ac8ae3de7701deb6749" exitCode=0 Oct 30 21:59:39.683255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.683256 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" event={"ID":"3836a17c-9eeb-4bd0-bd17-8f77fc66e340","Type":"ContainerDied","Data":"adb2383ef084693422561370d2c21bc5f891dfa510f33ac8ae3de7701deb6749"} Oct 30 21:59:39.855905 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.855869 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 21:59:39.860625 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.860609 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:39.867301 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.867277 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c614f9\"" Oct 30 21:59:39.868435 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.867873 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c614f9-dockercfg-zls4k\"" Oct 30 21:59:39.877682 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.877654 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 21:59:39.885616 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.885592 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 21:59:39.891494 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.891474 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:39.918849 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.918823 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 21:59:39.977346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.974927 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmdk\" (UniqueName: \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:39.977346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.975238 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9w5h\" (UniqueName: \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:39.977346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.975400 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:39.977346 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:39.975440 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.183910 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.183861 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9w5h\" (UniqueName: \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.186393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.184189 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:40.186393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.184241 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.186393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.184495 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmdk\" (UniqueName: \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:40.186393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.186290 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:40.186393 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.186254 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.199772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.199737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9w5h\" (UniqueName: \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\") pod \"payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.201471 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.201422 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmdk\" (UniqueName: \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\") pod \"payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:40.231203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.231125 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:40.480788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.480744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:40.590628 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.590598 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 21:59:40.592062 ip-10-0-133-193 kubenswrapper[5036]: W1030 21:59:40.592037 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f0b86_70df_43b9_806b_201e32fc8d36.slice/crio-e0d1a1a96eb65ef98fa126b0fe8d226c21f1e76f5cfeea9bba046231fed7028b WatchSource:0}: Error finding container e0d1a1a96eb65ef98fa126b0fe8d226c21f1e76f5cfeea9bba046231fed7028b: Status 404 returned error can't find the container with id e0d1a1a96eb65ef98fa126b0fe8d226c21f1e76f5cfeea9bba046231fed7028b Oct 30 21:59:40.695661 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.695630 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" event={"ID":"714f0b86-70df-43b9-806b-201e32fc8d36","Type":"ContainerStarted","Data":"c45a41b3a2150b8924f701c3dd71255c0e0ca3dd3f8f611fc34f6ddebe8d0f16"} Oct 30 21:59:40.696008 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.695696 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" event={"ID":"714f0b86-70df-43b9-806b-201e32fc8d36","Type":"ContainerStarted","Data":"e0d1a1a96eb65ef98fa126b0fe8d226c21f1e76f5cfeea9bba046231fed7028b"} Oct 30 21:59:40.734607 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.734549 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" podStartSLOduration=1.734532693 podStartE2EDuration="1.734532693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:40.723191874 +0000 UTC m=+1204.673146820" watchObservedRunningTime="2026-10-30 21:59:40.734532693 +0000 UTC m=+1204.684487639" Oct 30 21:59:40.808534 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.808500 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 21:59:40.926549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:40.926505 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:41.010987 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.008999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\") pod \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " Oct 30 21:59:41.010987 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.009306 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\" (UniqueName: \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\") pod \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " Oct 30 21:59:41.010987 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.009463 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\") pod \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\" (UID: \"3836a17c-9eeb-4bd0-bd17-8f77fc66e340\") " Oct 30 21:59:41.021559 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.020503 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca" pod "3836a17c-9eeb-4bd0-bd17-8f77fc66e340" (UID: "3836a17c-9eeb-4bd0-bd17-8f77fc66e340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:41.127292 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.126928 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:41.177293 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.168078 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca" pod "3836a17c-9eeb-4bd0-bd17-8f77fc66e340" (UID: "3836a17c-9eeb-4bd0-bd17-8f77fc66e340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:41.199531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.199480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x" pod "3836a17c-9eeb-4bd0-bd17-8f77fc66e340" (UID: "3836a17c-9eeb-4bd0-bd17-8f77fc66e340"). InnerVolumeSpecName "kube-api-access-k965x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:41.250900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.241433 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:41.250900 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.241468 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k965x\" (UniqueName: \"kubernetes.io/projected/3836a17c-9eeb-4bd0-bd17-8f77fc66e340-kube-api-access-k965x\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:41.338400 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.338320 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9gpsn_fed55f70-e371-4b9a-9258-e70a781e919c/payload-processing/0.log" Oct 30 21:59:41.560476 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.560450 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9gpsn_fed55f70-e371-4b9a-9258-e70a781e919c/payload-processing/0.log" Oct 30 21:59:41.705772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.705736 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" event={"ID":"3b6bac20-909d-4354-9f53-6cd63e65a781","Type":"ContainerStarted","Data":"d62ac5588cb0b6e90ddbe396498711e00e8246f1c86c992eb10c3681778a1896"} Oct 30 21:59:41.706255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.705823 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" event={"ID":"3b6bac20-909d-4354-9f53-6cd63e65a781","Type":"ContainerStarted","Data":"41f6ec5b6f58e3aaacd57e029b558bb18bb82d5e6309d990d296b61638bc59f9"} Oct 30 21:59:41.706255 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.705969 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:41.707410 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.707390 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" Oct 30 21:59:41.707531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.707423 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5" event={"ID":"3836a17c-9eeb-4bd0-bd17-8f77fc66e340","Type":"ContainerDied","Data":"af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee"} Oct 30 21:59:41.707531 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.707466 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af015508df33aeb715e48db53d09e2755d4714dd95139a782f9e57bf8cdbe2ee" Oct 30 21:59:41.707759 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.707741 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:41.731662 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.731615 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" podStartSLOduration=2.731601698 podStartE2EDuration="2.731601698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:41.726240523 +0000 UTC m=+1205.676195490" watchObservedRunningTime="2026-10-30 21:59:41.731601698 +0000 UTC m=+1205.681556644" Oct 30 21:59:41.748866 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:41.748838 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst_52b962ac-f06e-4e57-92a5-74248dcd4eb1/payload-processing/0.log" Oct 30 21:59:42.788772 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:42.788736 5036 generic.go:356] "Generic (PLEG): container finished" podID="a316ac0c-a489-466e-a974-40530fa15cae" containerID="3096fe16f82ed942b7e90e977d83018e2733621e9934218600693864bb1dda09" exitCode=0 Oct 30 21:59:42.790315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:42.790097 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerDied","Data":"3096fe16f82ed942b7e90e977d83018e2733621e9934218600693864bb1dda09"} Oct 30 21:59:42.790315 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:42.790184 5036 scope.go:122] "RemoveContainer" containerID="de8dc85eb7fc4ceaaeeb97e10b10c1f4a98ed796122ee5278d6be3acca77d408" Oct 30 21:59:43.015549 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.015519 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:43.088095 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088061 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088164 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088295 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088363 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088358 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.088599 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088537 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.089016 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088618 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.089016 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.088800 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\") pod \"a316ac0c-a489-466e-a974-40530fa15cae\" (UID: \"a316ac0c-a489-466e-a974-40530fa15cae\") " Oct 30 21:59:43.092598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.092208 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.092598 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.092547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.092784 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.092762 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:43.092870 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.092840 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.092922 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.092862 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.095609 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.095566 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.095874 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.095850 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "kube-api-access-9gv47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:43.096837 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.096802 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:43.097203 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.097180 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token" pod "a316ac0c-a489-466e-a974-40530fa15cae" (UID: "a316ac0c-a489-466e-a974-40530fa15cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:43.190846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190812 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.190846 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190845 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190861 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a316ac0c-a489-466e-a974-40530fa15cae-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190874 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190887 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190898 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a316ac0c-a489-466e-a974-40530fa15cae-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190912 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190925 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gv47\" (UniqueName: \"kubernetes.io/projected/a316ac0c-a489-466e-a974-40530fa15cae-kube-api-access-9gv47\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.191083 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.190938 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a316ac0c-a489-466e-a974-40530fa15cae-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.801142 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.801106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" event={"ID":"a316ac0c-a489-466e-a974-40530fa15cae","Type":"ContainerDied","Data":"176398b9bb4c3e62532a7c78340194be84dd197103fc6bfc2ae6330fd2573480"} Oct 30 21:59:43.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.801161 5036 scope.go:122] "RemoveContainer" containerID="3096fe16f82ed942b7e90e977d83018e2733621e9934218600693864bb1dda09" Oct 30 21:59:43.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.801247 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2" Oct 30 21:59:43.803245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.803219 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr"] Oct 30 21:59:43.804232 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804211 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:43.804373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804238 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:43.804373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804282 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:43.804373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804290 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:43.804373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804333 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3836a17c-9eeb-4bd0-bd17-8f77fc66e340" containerName="revoke-keys" Oct 30 21:59:43.804373 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804341 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3836a17c-9eeb-4bd0-bd17-8f77fc66e340" containerName="revoke-keys" Oct 30 21:59:43.804635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804598 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:43.804635 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.804620 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3836a17c-9eeb-4bd0-bd17-8f77fc66e340" containerName="revoke-keys" Oct 30 21:59:43.811030 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.810366 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:43.815496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.815178 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:43.815496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.815336 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:43.815496 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.815428 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:59:43.815976 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.815959 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:43.837880 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.837856 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr"] Oct 30 21:59:43.964289 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.962255 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:59:43.969060 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:43.969031 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-004c17-openshift-default-7bb5cb999f-7h2d2"] Oct 30 21:59:44.009890 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.009858 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.010105 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.010082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4km\" (UniqueName: \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.010178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.010119 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.048214 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.048135 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc"] Oct 30 21:59:44.058468 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.058341 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq"] Oct 30 21:59:44.061906 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.061877 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6a354b-2qqmc"] Oct 30 21:59:44.069788 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.069762 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f33970-4mnqq"] Oct 30 21:59:44.111956 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.111925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.112179 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.112161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4km\" (UniqueName: \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.112245 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.112200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.113020 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.112991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.114685 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.114664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.125691 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.125668 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4km\" (UniqueName: \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\") pod \"maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.195578 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.195548 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264c4133-0568-4128-b34f-c1db323b012a" path="/var/lib/kubelet/pods/264c4133-0568-4128-b34f-c1db323b012a/volumes" Oct 30 21:59:44.196069 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.196054 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a316ac0c-a489-466e-a974-40530fa15cae" path="/var/lib/kubelet/pods/a316ac0c-a489-466e-a974-40530fa15cae/volumes" Oct 30 21:59:44.196624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.196597 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9f54d1-e72d-431d-af6c-db7f63e13534" path="/var/lib/kubelet/pods/aa9f54d1-e72d-431d-af6c-db7f63e13534/volumes" Oct 30 21:59:44.239711 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.239687 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:44.537299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.537235 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr"] Oct 30 21:59:44.813009 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:44.812962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" event={"ID":"399df661-1f8e-4bde-8d63-5128243aacda","Type":"ContainerStarted","Data":"8e1276a6db4af6f78b50b45843ee3ef5bfe305e6347301838b9eaca0822ecaa3"} Oct 30 21:59:45.829561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:45.829447 5036 generic.go:356] "Generic (PLEG): container finished" podID="399df661-1f8e-4bde-8d63-5128243aacda" containerID="86b4b9f9c1aa90a010f4da39393c56823ab074863ea7d5f6b1d78eee72f89c5a" exitCode=0 Oct 30 21:59:45.829561 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:45.829531 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" event={"ID":"399df661-1f8e-4bde-8d63-5128243aacda","Type":"ContainerDied","Data":"86b4b9f9c1aa90a010f4da39393c56823ab074863ea7d5f6b1d78eee72f89c5a"} Oct 30 21:59:47.042042 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.042011 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9"] Oct 30 21:59:47.049124 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.049097 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92926d6-dc6v9"] Oct 30 21:59:47.053499 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.053476 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:47.084756 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.084712 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\") pod \"399df661-1f8e-4bde-8d63-5128243aacda\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " Oct 30 21:59:47.084958 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.084808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\" (UniqueName: \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\") pod \"399df661-1f8e-4bde-8d63-5128243aacda\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " Oct 30 21:59:47.088342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.085085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\") pod \"399df661-1f8e-4bde-8d63-5128243aacda\" (UID: \"399df661-1f8e-4bde-8d63-5128243aacda\") " Oct 30 21:59:47.088342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.086398 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca" pod "399df661-1f8e-4bde-8d63-5128243aacda" (UID: "399df661-1f8e-4bde-8d63-5128243aacda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:47.088342 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.087072 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:47.107747 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.107504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km" pod "399df661-1f8e-4bde-8d63-5128243aacda" (UID: "399df661-1f8e-4bde-8d63-5128243aacda"). InnerVolumeSpecName "kube-api-access-rw4km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:47.148954 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.147446 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca" pod "399df661-1f8e-4bde-8d63-5128243aacda" (UID: "399df661-1f8e-4bde-8d63-5128243aacda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:47.189038 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.189000 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399df661-1f8e-4bde-8d63-5128243aacda-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:47.189216 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.189043 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4km\" (UniqueName: \"kubernetes.io/projected/399df661-1f8e-4bde-8d63-5128243aacda-kube-api-access-rw4km\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:47.744683 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.744645 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:59:47.764493 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.764463 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:59:47.851798 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.851775 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" Oct 30 21:59:47.852055 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.851813 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr" event={"ID":"399df661-1f8e-4bde-8d63-5128243aacda","Type":"ContainerDied","Data":"8e1276a6db4af6f78b50b45843ee3ef5bfe305e6347301838b9eaca0822ecaa3"} Oct 30 21:59:47.852146 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:47.852120 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e1276a6db4af6f78b50b45843ee3ef5bfe305e6347301838b9eaca0822ecaa3" Oct 30 21:59:48.246299 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:48.244248 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842439b9-b906-46bc-8179-38dd22461bed" path="/var/lib/kubelet/pods/842439b9-b906-46bc-8179-38dd22461bed/volumes" Oct 30 21:59:48.295294 ip-10-0-133-193 kubenswrapper[5036]: E1030 21:59:48.294832 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 21:59:49.686459 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:49.686432 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 21:59:49.686872 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:49.686498 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 21:59:52.745422 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.745379 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" podUID="3fec8243-a064-4c9d-920a-7e960ba0c790" containerName="payload-processing" containerID="cri-o://62afcff1028b1dd1f82d2fc145fb8c098728d2373592055aa0f3ed60538a8464" gracePeriod=25 Oct 30 21:59:52.765129 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.765091 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" podUID="eb855346-a630-40f7-b32b-91dccdc68205" containerName="payload-pre-processing" containerID="cri-o://7b3d7be8363c74c98505c1e27db9174ed27d8cfec9cdfe18a6a0f5e2a8427377" gracePeriod=25 Oct 30 21:59:52.791763 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.791741 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 21:59:52.795715 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.795697 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 21:59:52.921286 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.921231 5036 generic.go:356] "Generic (PLEG): container finished" podID="3fec8243-a064-4c9d-920a-7e960ba0c790" containerID="62afcff1028b1dd1f82d2fc145fb8c098728d2373592055aa0f3ed60538a8464" exitCode=0 Oct 30 21:59:52.921478 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.921447 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" event={"ID":"3fec8243-a064-4c9d-920a-7e960ba0c790","Type":"ContainerDied","Data":"62afcff1028b1dd1f82d2fc145fb8c098728d2373592055aa0f3ed60538a8464"} Oct 30 21:59:52.923478 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.923453 5036 generic.go:356] "Generic (PLEG): container finished" podID="eb855346-a630-40f7-b32b-91dccdc68205" containerID="7b3d7be8363c74c98505c1e27db9174ed27d8cfec9cdfe18a6a0f5e2a8427377" exitCode=0 Oct 30 21:59:52.923633 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:52.923518 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" event={"ID":"eb855346-a630-40f7-b32b-91dccdc68205","Type":"ContainerDied","Data":"7b3d7be8363c74c98505c1e27db9174ed27d8cfec9cdfe18a6a0f5e2a8427377"} Oct 30 21:59:53.150178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.150145 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:59:53.209554 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.209524 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:59:53.244592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.243135 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\") pod \"3fec8243-a064-4c9d-920a-7e960ba0c790\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " Oct 30 21:59:53.244592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.243388 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\" (UniqueName: \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\") pod \"3fec8243-a064-4c9d-920a-7e960ba0c790\" (UID: \"3fec8243-a064-4c9d-920a-7e960ba0c790\") " Oct 30 21:59:53.244592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.243873 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume" pod "3fec8243-a064-4c9d-920a-7e960ba0c790" (UID: "3fec8243-a064-4c9d-920a-7e960ba0c790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:53.244592 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.244213 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec8243-a064-4c9d-920a-7e960ba0c790-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.250442 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.250412 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb" pod "3fec8243-a064-4c9d-920a-7e960ba0c790" (UID: "3fec8243-a064-4c9d-920a-7e960ba0c790"). InnerVolumeSpecName "kube-api-access-265lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:53.345178 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.345085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\") pod \"eb855346-a630-40f7-b32b-91dccdc68205\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " Oct 30 21:59:53.345503 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.345468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\" (UniqueName: \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\") pod \"eb855346-a630-40f7-b32b-91dccdc68205\" (UID: \"eb855346-a630-40f7-b32b-91dccdc68205\") " Oct 30 21:59:53.345586 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.345517 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume" pod "eb855346-a630-40f7-b32b-91dccdc68205" (UID: "eb855346-a630-40f7-b32b-91dccdc68205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:53.346238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.346204 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265lb\" (UniqueName: \"kubernetes.io/projected/3fec8243-a064-4c9d-920a-7e960ba0c790-kube-api-access-265lb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.346238 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.346230 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb855346-a630-40f7-b32b-91dccdc68205-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.347856 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.347835 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z" pod "eb855346-a630-40f7-b32b-91dccdc68205" (UID: "eb855346-a630-40f7-b32b-91dccdc68205"). InnerVolumeSpecName "kube-api-access-z7h7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:53.448256 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.448221 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7h7z\" (UniqueName: \"kubernetes.io/projected/eb855346-a630-40f7-b32b-91dccdc68205-kube-api-access-z7h7z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.936953 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.936919 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" Oct 30 21:59:53.937507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.936951 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw" event={"ID":"3fec8243-a064-4c9d-920a-7e960ba0c790","Type":"ContainerDied","Data":"b8d46344662e1e30eff415510119af88440bea25c7fcbba30c95a64ee10412dd"} Oct 30 21:59:53.937507 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.937020 5036 scope.go:122] "RemoveContainer" containerID="62afcff1028b1dd1f82d2fc145fb8c098728d2373592055aa0f3ed60538a8464" Oct 30 21:59:53.946492 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.946473 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" Oct 30 21:59:53.946628 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.946570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s" event={"ID":"eb855346-a630-40f7-b32b-91dccdc68205","Type":"ContainerDied","Data":"964326264dba5653e54ef8d2805d3067221bfad401de7e92f916feda8f9c3f02"} Oct 30 21:59:53.956885 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.956858 5036 scope.go:122] "RemoveContainer" containerID="7b3d7be8363c74c98505c1e27db9174ed27d8cfec9cdfe18a6a0f5e2a8427377" Oct 30 21:59:53.964058 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.964033 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:59:53.968654 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.968630 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f47f79-6fc7759f6c-mgvxw"] Oct 30 21:59:53.979177 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.979150 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:59:53.987306 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:53.987283 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f47f79-64fd776bcf-zqr6s"] Oct 30 21:59:54.194563 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:54.194518 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fec8243-a064-4c9d-920a-7e960ba0c790" path="/var/lib/kubelet/pods/3fec8243-a064-4c9d-920a-7e960ba0c790/volumes" Oct 30 21:59:54.195070 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:54.195046 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb855346-a630-40f7-b32b-91dccdc68205" path="/var/lib/kubelet/pods/eb855346-a630-40f7-b32b-91dccdc68205/volumes" Oct 30 21:59:58.065762 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.065730 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 21:59:58.066823 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066760 5036 cpu_manager.go:414] "containerMap: removing container" podUID="399df661-1f8e-4bde-8d63-5128243aacda" containerName="revoke-keys" Oct 30 21:59:58.066823 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066784 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399df661-1f8e-4bde-8d63-5128243aacda" containerName="revoke-keys" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066860 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3fec8243-a064-4c9d-920a-7e960ba0c790" containerName="payload-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066870 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fec8243-a064-4c9d-920a-7e960ba0c790" containerName="payload-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066907 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eb855346-a630-40f7-b32b-91dccdc68205" containerName="payload-pre-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.066915 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb855346-a630-40f7-b32b-91dccdc68205" containerName="payload-pre-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.067057 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a316ac0c-a489-466e-a974-40530fa15cae" containerName="istio-proxy" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.067079 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fec8243-a064-4c9d-920a-7e960ba0c790" containerName="payload-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.067091 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb855346-a630-40f7-b32b-91dccdc68205" containerName="payload-pre-processing" Oct 30 21:59:58.068513 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.067101 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="399df661-1f8e-4bde-8d63-5128243aacda" containerName="revoke-keys" Oct 30 21:59:58.073136 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.073112 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080378 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080509 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080393 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080589 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080663 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080591 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfj9\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080729 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080670 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080857 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080831 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.080989 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.080969 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.081138 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.081119 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.081217 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.081169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.091190 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.091160 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e5289f-openshift-default-dockercfg-7wjh7\"" Oct 30 21:59:58.096450 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.096421 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362237 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362323 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362435 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfj9\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362603 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362727 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.366337 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.362897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.500733 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.500644 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.554385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.502327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.554385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.503857 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.554385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.504229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.554385 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.543823 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.555977 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.555935 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.556873 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.556838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.589333 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.569220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.589624 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.572333 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfj9\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\") pod \"e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:58.703998 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:58.703967 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 21:59:59.051727 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:59.051697 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 21:59:59.068226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:59.067615 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:59.068226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:59.067761 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:59.068226 ip-10-0-133-193 kubenswrapper[5036]: I1030 21:59:59.067849 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:00.065957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.065916 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerStarted","Data":"5689143d0cdb33a710fa4193c5616452bac9087765a96201bd880356e92e78af"} Oct 30 22:00:00.066180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.065968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerStarted","Data":"29dce98df894ddffd21a7e6149445d983955c63504698b67ffff00fb90faef48"} Oct 30 22:00:00.098784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.098723 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podStartSLOduration=3.098705679 podStartE2EDuration="3.098705679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:00.096148317 +0000 UTC m=+1224.046103264" watchObservedRunningTime="2026-10-30 22:00:00.098705679 +0000 UTC m=+1224.048660626" Oct 30 22:00:00.282364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.282322 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.337821 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.338468 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.373057 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.373783 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.375993 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.396713 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:00.402095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.396818 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.431601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.430293 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.431601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.430406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.431601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.430447 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwm4\" (UniqueName: \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.431601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.430645 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5zg\" (UniqueName: \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.450957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.450928 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:00.451907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.451882 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.466732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.465696 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.466732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.465589 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:00.472515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.472447 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:00.472515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.472492 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:00.472515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.472520 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:00.472748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.472557 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.475783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.475765 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:00.483752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.483460 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:00.498308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.498225 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:00.504898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.503561 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.539290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.539207 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.544978 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27nfz\" (UniqueName: \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545015 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmhf\" (UniqueName: \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545033 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6gl\" (UniqueName: \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545054 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545099 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545139 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pqw\" (UniqueName: \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545175 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545208 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.545346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545231 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwm4\" (UniqueName: \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.546008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545425 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.546008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.545473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5zg\" (UniqueName: \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.546774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.546590 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.546774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.546608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.551070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.551049 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:00.551193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.551175 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.557507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.557488 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:00.557683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.557668 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.563399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.563383 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:00.563506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.563492 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.563676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.563526 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:00.568852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.568833 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:00.576335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.576308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5zg\" (UniqueName: \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\") pod \"maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.576536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.576515 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwm4\" (UniqueName: \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\") pod \"maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.599152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.599111 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:00.646917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.646868 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4blr\" (UniqueName: \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.647221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27nfz\" (UniqueName: \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.647409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmhf\" (UniqueName: \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.647542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647528 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6gl\" (UniqueName: \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.647663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647649 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.647864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pqw\" (UniqueName: \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.647999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.647986 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m87\" (UniqueName: \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.648121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.648219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648206 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.648428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.648553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.648716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw8w\" (UniqueName: \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.648825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648813 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.648986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648973 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.649168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.649146 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.649235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.648716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.649928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.649907 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.650533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.650511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.663069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.662986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27nfz\" (UniqueName: \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.663164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.663068 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmhf\" (UniqueName: \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\") pod \"maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.663661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.663645 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6gl\" (UniqueName: \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.663998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.663981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pqw\" (UniqueName: \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.688205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.688180 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:00.704161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.704134 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:00.711724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.711364 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:00.760323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.760288 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw8w\" (UniqueName: \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.760939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.760915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.763099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.762892 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:00.763234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.761902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.763501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.763483 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4blr\" (UniqueName: \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.764102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.764083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m87\" (UniqueName: \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.764612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.764592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.765555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.765536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.767029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.765476 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.769479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.767190 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.771761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.771740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:00.774548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.774479 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw8w\" (UniqueName: \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.775656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.775642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4blr\" (UniqueName: \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\") pod \"maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.776145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.776104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m87\" (UniqueName: \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\") pod \"maas-api-key-cleanup-29889960-wznq9\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:00.784565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.784541 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:00.798727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.798705 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:00.857474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.857432 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:00.874509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.873729 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:00.885054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.885021 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 22:00:00.904294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.899142 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 22:00:00.910324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.910092 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:00.957282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:00.946331 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:01.093360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.086107 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:01.112105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.112073 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:01.127015 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:01.120023 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee085cb9_48f1_4cc3_bc3d_b5649bf69064.slice/crio-186d7a6dc069cb260b5e5e7d90edb730bb001f8a613031b930c37941a58a53e7 WatchSource:0}: Error finding container 186d7a6dc069cb260b5e5e7d90edb730bb001f8a613031b930c37941a58a53e7: Status 404 returned error can't find the container with id 186d7a6dc069cb260b5e5e7d90edb730bb001f8a613031b930c37941a58a53e7 Oct 30 22:00:01.127015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.121205 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:01.699464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.699437 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:01.716216 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:01.716179 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4207f3dc_e5c9_4a91_9e71_34b83533046c.slice/crio-019627e378f6147ae230e6c7b0bfc21bbcb4edf6aebc497246740075f5dfc4d4 WatchSource:0}: Error finding container 019627e378f6147ae230e6c7b0bfc21bbcb4edf6aebc497246740075f5dfc4d4: Status 404 returned error can't find the container with id 019627e378f6147ae230e6c7b0bfc21bbcb4edf6aebc497246740075f5dfc4d4 Oct 30 22:00:01.741506 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:01.741437 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa478d93_fc7a_4a9b_a3dc_469524500757.slice/crio-91bfa0b87cda7b84efd3a6ad9f8504f59cb918a743c822b8236b684c3ccf788c WatchSource:0}: Error finding container 91bfa0b87cda7b84efd3a6ad9f8504f59cb918a743c822b8236b684c3ccf788c: Status 404 returned error can't find the container with id 91bfa0b87cda7b84efd3a6ad9f8504f59cb918a743c822b8236b684c3ccf788c Oct 30 22:00:01.765498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.765410 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:01.781076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.781038 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:01.852683 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:01.852207 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a815ffc_c527_4219_a8dc_518b21c2f4cc.slice/crio-cc76ecd221af6103ef5fdddfd9b4edfe05c695f31c26a1d0e7a5c096d6f0de60 WatchSource:0}: Error finding container cc76ecd221af6103ef5fdddfd9b4edfe05c695f31c26a1d0e7a5c096d6f0de60: Status 404 returned error can't find the container with id cc76ecd221af6103ef5fdddfd9b4edfe05c695f31c26a1d0e7a5c096d6f0de60 Oct 30 22:00:01.867853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.864388 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:01.871789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.871755 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:01.905634 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:01.905604 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8e006c_38db_4d8f_abe6_dbc14851195b.slice/crio-994b5a7c0e7213c147a4606f854a2c4568c71deca97314272bbc64e328ad4bdb WatchSource:0}: Error finding container 994b5a7c0e7213c147a4606f854a2c4568c71deca97314272bbc64e328ad4bdb: Status 404 returned error can't find the container with id 994b5a7c0e7213c147a4606f854a2c4568c71deca97314272bbc64e328ad4bdb Oct 30 22:00:01.918535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.918484 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:01.927483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.927463 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:01.995626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:01.995593 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:02.020803 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:02.020764 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf284e515_5326_487f_8c53_f7c87506bae2.slice/crio-a522e3a24186ae241488e49d524a653c0358c6d7cca6c583c329eea4f5cbc986 WatchSource:0}: Error finding container a522e3a24186ae241488e49d524a653c0358c6d7cca6c583c329eea4f5cbc986: Status 404 returned error can't find the container with id a522e3a24186ae241488e49d524a653c0358c6d7cca6c583c329eea4f5cbc986 Oct 30 22:00:02.129976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.129950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerStarted","Data":"a522e3a24186ae241488e49d524a653c0358c6d7cca6c583c329eea4f5cbc986"} Oct 30 22:00:02.132172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.132115 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerStarted","Data":"3cfeaf4bcf20252bd0928ff2501717a7996118ef3348a0e9a3eb03b5ee3ba51d"} Oct 30 22:00:02.133699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.133673 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerStarted","Data":"4dc0beebd3f3bfc51e895d50822d34bdf30e7dcc70342caf4fb32064d54e497b"} Oct 30 22:00:02.135359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.135337 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerStarted","Data":"09f4f91ee5c478818efa3f2f6ae149fbc071d34211b33304c8c96cb9e2a0d0d1"} Oct 30 22:00:02.135458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.135382 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerStarted","Data":"186d7a6dc069cb260b5e5e7d90edb730bb001f8a613031b930c37941a58a53e7"} Oct 30 22:00:02.136878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.136857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerStarted","Data":"019627e378f6147ae230e6c7b0bfc21bbcb4edf6aebc497246740075f5dfc4d4"} Oct 30 22:00:02.138190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.138171 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerStarted","Data":"91bfa0b87cda7b84efd3a6ad9f8504f59cb918a743c822b8236b684c3ccf788c"} Oct 30 22:00:02.139691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.139669 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerStarted","Data":"994b5a7c0e7213c147a4606f854a2c4568c71deca97314272bbc64e328ad4bdb"} Oct 30 22:00:02.141177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.141154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerStarted","Data":"cc76ecd221af6103ef5fdddfd9b4edfe05c695f31c26a1d0e7a5c096d6f0de60"} Oct 30 22:00:02.142643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.142622 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerStarted","Data":"8d658244ce682ebca31601b50757d239bdb5aa2e3a5a047a79dfcd38411cb3d9"} Oct 30 22:00:02.160330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:02.160259 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" podStartSLOduration=2.1602411679999998 podStartE2EDuration="2.160241168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.15348685 +0000 UTC m=+1226.103441796" watchObservedRunningTime="2026-10-30 22:00:02.160241168 +0000 UTC m=+1226.110196118" Oct 30 22:00:03.082299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.078360 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl"] Oct 30 22:00:03.085052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.085009 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd443a01-4lbgl"] Oct 30 22:00:03.177801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.177770 5036 generic.go:356] "Generic (PLEG): container finished" podID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerID="09f4f91ee5c478818efa3f2f6ae149fbc071d34211b33304c8c96cb9e2a0d0d1" exitCode=60 Oct 30 22:00:03.178365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.178044 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerDied","Data":"09f4f91ee5c478818efa3f2f6ae149fbc071d34211b33304c8c96cb9e2a0d0d1"} Oct 30 22:00:03.178365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.178335 5036 scope.go:122] "RemoveContainer" containerID="09f4f91ee5c478818efa3f2f6ae149fbc071d34211b33304c8c96cb9e2a0d0d1" Oct 30 22:00:03.180648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.180483 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerStarted","Data":"9941d626f27ac5ee0bb0a21ef045c88ef086e4f65f406f94d493b3f55a95b2ab"} Oct 30 22:00:03.180701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.180691 5036 scope.go:122] "RemoveContainer" containerID="9941d626f27ac5ee0bb0a21ef045c88ef086e4f65f406f94d493b3f55a95b2ab" Oct 30 22:00:03.186729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.184021 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerStarted","Data":"a41ba985c288fff2d1c132d129ce6fb614b9aa8237595b5263b6d035bf20d492"} Oct 30 22:00:03.186729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.186600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerStarted","Data":"3b8cdd297b8b59e3e023e7a2e4ceacd82b6d45c16d5524e27576e81cb9192f93"} Oct 30 22:00:03.188564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.188537 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerID="c4375a28e0a8078a3edef7084deb9e523021879ce1afecb6aac0deab5b068f59" exitCode=60 Oct 30 22:00:03.188672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.188621 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerDied","Data":"c4375a28e0a8078a3edef7084deb9e523021879ce1afecb6aac0deab5b068f59"} Oct 30 22:00:03.188806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.188785 5036 scope.go:122] "RemoveContainer" containerID="c4375a28e0a8078a3edef7084deb9e523021879ce1afecb6aac0deab5b068f59" Oct 30 22:00:03.191436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.190807 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerStarted","Data":"b689bbccacf05b7363d1fa1526186c9ccb8115ead2606d35c2302271561cfcbe"} Oct 30 22:00:03.191436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.191077 5036 scope.go:122] "RemoveContainer" containerID="b689bbccacf05b7363d1fa1526186c9ccb8115ead2606d35c2302271561cfcbe" Oct 30 22:00:03.193779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.193443 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerStarted","Data":"32e972f82292cc402b9cb7202bee0c40e7e6132e88109b541b6920736476ae09"} Oct 30 22:00:03.196009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.195984 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerStarted","Data":"c35d3fcd6a9ce2b2fb3fa24dcd429112f6c57949ec32449545d1967bbb779d68"} Oct 30 22:00:03.198318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.197963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerStarted","Data":"3ec6fde6d7beb164308245a1dbde64bde02b8b599744e632b50065ed077410c2"} Oct 30 22:00:03.368703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.368654 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" podStartSLOduration=3.3686395080000002 podStartE2EDuration="3.368639508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.360945701 +0000 UTC m=+1227.310900647" watchObservedRunningTime="2026-10-30 22:00:03.368639508 +0000 UTC m=+1227.318594453" Oct 30 22:00:03.417387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.417211 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" podStartSLOduration=3.417193762 podStartE2EDuration="3.417193762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.412479125 +0000 UTC m=+1227.362434074" watchObservedRunningTime="2026-10-30 22:00:03.417193762 +0000 UTC m=+1227.367148709" Oct 30 22:00:03.454386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.454334 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" podStartSLOduration=3.454317825 podStartE2EDuration="3.454317825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.454066962 +0000 UTC m=+1227.404021909" watchObservedRunningTime="2026-10-30 22:00:03.454317825 +0000 UTC m=+1227.404272772" Oct 30 22:00:03.486255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.486208 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" podStartSLOduration=3.486194467 podStartE2EDuration="3.486194467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.484792244 +0000 UTC m=+1227.434747190" watchObservedRunningTime="2026-10-30 22:00:03.486194467 +0000 UTC m=+1227.436149412" Oct 30 22:00:03.546891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:03.546842 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" podStartSLOduration=3.546827261 podStartE2EDuration="3.546827261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.541623846 +0000 UTC m=+1227.491578817" watchObservedRunningTime="2026-10-30 22:00:03.546827261 +0000 UTC m=+1227.496782223" Oct 30 22:00:04.201706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.201652 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121ab45f-f276-4708-ac06-7f8acb92d05c" path="/var/lib/kubelet/pods/121ab45f-f276-4708-ac06-7f8acb92d05c/volumes" Oct 30 22:00:04.219793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.219754 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerStarted","Data":"d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4"} Oct 30 22:00:04.224989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.224960 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c1036b5-a293-4661-b033-7eab36c87224" containerID="b689bbccacf05b7363d1fa1526186c9ccb8115ead2606d35c2302271561cfcbe" exitCode=60 Oct 30 22:00:04.225125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.225106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerDied","Data":"b689bbccacf05b7363d1fa1526186c9ccb8115ead2606d35c2302271561cfcbe"} Oct 30 22:00:04.225169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.225146 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerStarted","Data":"4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a"} Oct 30 22:00:04.228056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.228032 5036 generic.go:356] "Generic (PLEG): container finished" podID="f284e515-5326-487f-8c53-f7c87506bae2" containerID="32e972f82292cc402b9cb7202bee0c40e7e6132e88109b541b6920736476ae09" exitCode=60 Oct 30 22:00:04.228196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.228170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerDied","Data":"32e972f82292cc402b9cb7202bee0c40e7e6132e88109b541b6920736476ae09"} Oct 30 22:00:04.228430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.228402 5036 scope.go:122] "RemoveContainer" containerID="32e972f82292cc402b9cb7202bee0c40e7e6132e88109b541b6920736476ae09" Oct 30 22:00:04.231147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.231123 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fcc2179-6acc-4de1-9573-75a113690029" containerID="c35d3fcd6a9ce2b2fb3fa24dcd429112f6c57949ec32449545d1967bbb779d68" exitCode=60 Oct 30 22:00:04.231324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.231307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerDied","Data":"c35d3fcd6a9ce2b2fb3fa24dcd429112f6c57949ec32449545d1967bbb779d68"} Oct 30 22:00:04.231602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.231586 5036 scope.go:122] "RemoveContainer" containerID="c35d3fcd6a9ce2b2fb3fa24dcd429112f6c57949ec32449545d1967bbb779d68" Oct 30 22:00:04.236236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.236212 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerID="3ec6fde6d7beb164308245a1dbde64bde02b8b599744e632b50065ed077410c2" exitCode=60 Oct 30 22:00:04.236359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.236336 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerDied","Data":"3ec6fde6d7beb164308245a1dbde64bde02b8b599744e632b50065ed077410c2"} Oct 30 22:00:04.239936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.239547 5036 scope.go:122] "RemoveContainer" containerID="3ec6fde6d7beb164308245a1dbde64bde02b8b599744e632b50065ed077410c2" Oct 30 22:00:04.241150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.241127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerStarted","Data":"8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699"} Oct 30 22:00:04.248707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.248680 5036 generic.go:356] "Generic (PLEG): container finished" podID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerID="9941d626f27ac5ee0bb0a21ef045c88ef086e4f65f406f94d493b3f55a95b2ab" exitCode=60 Oct 30 22:00:04.249456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.249347 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerDied","Data":"9941d626f27ac5ee0bb0a21ef045c88ef086e4f65f406f94d493b3f55a95b2ab"} Oct 30 22:00:04.249456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.249396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerStarted","Data":"ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee"} Oct 30 22:00:04.261982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.254558 5036 generic.go:356] "Generic (PLEG): container finished" podID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerID="a41ba985c288fff2d1c132d129ce6fb614b9aa8237595b5263b6d035bf20d492" exitCode=60 Oct 30 22:00:04.261982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.257931 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" podStartSLOduration=4.257914288 podStartE2EDuration="4.257914288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.251114027 +0000 UTC m=+1228.201068975" watchObservedRunningTime="2026-10-30 22:00:04.257914288 +0000 UTC m=+1228.207869235" Oct 30 22:00:04.261982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.259161 5036 scope.go:122] "RemoveContainer" containerID="a41ba985c288fff2d1c132d129ce6fb614b9aa8237595b5263b6d035bf20d492" Oct 30 22:00:04.261982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.261692 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerDied","Data":"a41ba985c288fff2d1c132d129ce6fb614b9aa8237595b5263b6d035bf20d492"} Oct 30 22:00:04.267576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.266437 5036 generic.go:356] "Generic (PLEG): container finished" podID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerID="3b8cdd297b8b59e3e023e7a2e4ceacd82b6d45c16d5524e27576e81cb9192f93" exitCode=60 Oct 30 22:00:04.267576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.266481 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerDied","Data":"3b8cdd297b8b59e3e023e7a2e4ceacd82b6d45c16d5524e27576e81cb9192f93"} Oct 30 22:00:04.267576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.266806 5036 scope.go:122] "RemoveContainer" containerID="3b8cdd297b8b59e3e023e7a2e4ceacd82b6d45c16d5524e27576e81cb9192f93" Oct 30 22:00:04.306684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.303002 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" podStartSLOduration=4.302984993 podStartE2EDuration="4.302984993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.291006692 +0000 UTC m=+1228.240961635" watchObservedRunningTime="2026-10-30 22:00:04.302984993 +0000 UTC m=+1228.252939942" Oct 30 22:00:04.667655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:04.667588 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" podStartSLOduration=4.667568112 podStartE2EDuration="4.667568112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.488095725 +0000 UTC m=+1228.438050672" watchObservedRunningTime="2026-10-30 22:00:04.667568112 +0000 UTC m=+1228.617523059" Oct 30 22:00:05.001768 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:05.001736 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee085cb9_48f1_4cc3_bc3d_b5649bf69064.slice/crio-8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:00:05.279881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.279841 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerStarted","Data":"920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428"} Oct 30 22:00:05.283079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.283038 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerStarted","Data":"ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e"} Oct 30 22:00:05.285588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.285556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerStarted","Data":"f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477"} Oct 30 22:00:05.288005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.287977 5036 generic.go:356] "Generic (PLEG): container finished" podID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerID="8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699" exitCode=60 Oct 30 22:00:05.288117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.288071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerDied","Data":"8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699"} Oct 30 22:00:05.288181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.288119 5036 scope.go:122] "RemoveContainer" containerID="09f4f91ee5c478818efa3f2f6ae149fbc071d34211b33304c8c96cb9e2a0d0d1" Oct 30 22:00:05.288513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.288489 5036 scope.go:122] "RemoveContainer" containerID="8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699" Oct 30 22:00:05.288907 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:05.288880 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484_odh-ai-gateway-infra(ee085cb9-48f1-4cc3-bc3d-b5649bf69064)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" Oct 30 22:00:05.290367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.290344 5036 generic.go:356] "Generic (PLEG): container finished" podID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerID="ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee" exitCode=60 Oct 30 22:00:05.290850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.290463 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerDied","Data":"ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee"} Oct 30 22:00:05.290940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.290775 5036 scope.go:122] "RemoveContainer" containerID="ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee" Oct 30 22:00:05.291301 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:05.291241 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz_odh-ai-gateway-infra(4207f3dc-e5c9-4a91-9e71-34b83533046c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" Oct 30 22:00:05.293308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.293247 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerStarted","Data":"c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed"} Oct 30 22:00:05.297067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.296568 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerStarted","Data":"5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951"} Oct 30 22:00:05.299059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.299032 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerID="d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4" exitCode=60 Oct 30 22:00:05.299152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.299128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerDied","Data":"d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4"} Oct 30 22:00:05.299467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.299449 5036 scope.go:122] "RemoveContainer" containerID="d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4" Oct 30 22:00:05.299850 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:05.299758 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wznq9_odh-ai-gateway-infra(8a815ffc-c527-4219-a8dc-518b21c2f4cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" Oct 30 22:00:05.301578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.301555 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c1036b5-a293-4661-b033-7eab36c87224" containerID="4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a" exitCode=60 Oct 30 22:00:05.301699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.301641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerDied","Data":"4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a"} Oct 30 22:00:05.301961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.301942 5036 scope.go:122] "RemoveContainer" containerID="4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a" Oct 30 22:00:05.302312 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:05.302286 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf_odh-ai-gateway-infra(8c1036b5-a293-4661-b033-7eab36c87224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" podUID="8c1036b5-a293-4661-b033-7eab36c87224" Oct 30 22:00:05.306800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.306774 5036 scope.go:122] "RemoveContainer" containerID="9941d626f27ac5ee0bb0a21ef045c88ef086e4f65f406f94d493b3f55a95b2ab" Oct 30 22:00:05.350257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.350233 5036 scope.go:122] "RemoveContainer" containerID="c4375a28e0a8078a3edef7084deb9e523021879ce1afecb6aac0deab5b068f59" Oct 30 22:00:05.377094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.377041 5036 scope.go:122] "RemoveContainer" containerID="b689bbccacf05b7363d1fa1526186c9ccb8115ead2606d35c2302271561cfcbe" Oct 30 22:00:05.886065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.886007 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" podUID="4d8e343e-dbc7-4866-beec-2651bf37a636" containerName="payload-processing" containerID="cri-o://839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0" gracePeriod=25 Oct 30 22:00:05.900245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:05.900208 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" podUID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" containerName="payload-pre-processing" containerID="cri-o://69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1" gracePeriod=25 Oct 30 22:00:06.235609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.235533 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 22:00:06.276437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.276399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\") pod \"4d8e343e-dbc7-4866-beec-2651bf37a636\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " Oct 30 22:00:06.276759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.276718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\" (UniqueName: \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\") pod \"4d8e343e-dbc7-4866-beec-2651bf37a636\" (UID: \"4d8e343e-dbc7-4866-beec-2651bf37a636\") " Oct 30 22:00:06.277340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.277309 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume" pod "4d8e343e-dbc7-4866-beec-2651bf37a636" (UID: "4d8e343e-dbc7-4866-beec-2651bf37a636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:06.277595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.277571 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8e343e-dbc7-4866-beec-2651bf37a636-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:06.280353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.280330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x" pod "4d8e343e-dbc7-4866-beec-2651bf37a636" (UID: "4d8e343e-dbc7-4866-beec-2651bf37a636"). InnerVolumeSpecName "kube-api-access-rdj2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:06.285228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.285202 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 22:00:06.330187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.330149 5036 generic.go:356] "Generic (PLEG): container finished" podID="4d8e343e-dbc7-4866-beec-2651bf37a636" containerID="839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0" exitCode=0 Oct 30 22:00:06.330381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.330320 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" event={"ID":"4d8e343e-dbc7-4866-beec-2651bf37a636","Type":"ContainerDied","Data":"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0"} Oct 30 22:00:06.330381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.330378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" event={"ID":"4d8e343e-dbc7-4866-beec-2651bf37a636","Type":"ContainerDied","Data":"82aae333c8162f6e718fba308a92dce81723fb5df4d3cb14bb4e311c94e646b6"} Oct 30 22:00:06.330505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.330410 5036 scope.go:122] "RemoveContainer" containerID="839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0" Oct 30 22:00:06.330748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.330727 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg" Oct 30 22:00:06.343660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.343608 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerID="f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477" exitCode=60 Oct 30 22:00:06.344193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.343918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerDied","Data":"f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477"} Oct 30 22:00:06.344326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.344299 5036 scope.go:122] "RemoveContainer" containerID="f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477" Oct 30 22:00:06.345212 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.344749 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw_odh-ai-gateway-infra(c5d9a1be-eddc-4de6-b600-1407aada133b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" Oct 30 22:00:06.355282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.354980 5036 scope.go:122] "RemoveContainer" containerID="8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699" Oct 30 22:00:06.355443 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.355413 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484_odh-ai-gateway-infra(ee085cb9-48f1-4cc3-bc3d-b5649bf69064)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" Oct 30 22:00:06.365396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.365094 5036 scope.go:122] "RemoveContainer" containerID="839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0" Oct 30 22:00:06.365729 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.365658 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0\": container with ID starting with 839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0 not found: ID does not exist" containerID="839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0" Oct 30 22:00:06.365794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.365742 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0"} err="failed to get container status \"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0\": rpc error: code = NotFound desc = could not find container \"839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0\": container with ID starting with 839803265dae984ef12d8e2af3ab78a17a8e7aa701a01b4aa07a4e0b0bb283a0 not found: ID does not exist" Oct 30 22:00:06.365859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.365791 5036 scope.go:122] "RemoveContainer" containerID="3ec6fde6d7beb164308245a1dbde64bde02b8b599744e632b50065ed077410c2" Oct 30 22:00:06.383323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.383146 5036 scope.go:122] "RemoveContainer" containerID="ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.393746 5036 generic.go:356] "Generic (PLEG): container finished" podID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerID="c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed" exitCode=60 Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.394577 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\") pod \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.394949 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\" (UniqueName: \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\") pod \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\" (UID: \"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4\") " Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.400898 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz_odh-ai-gateway-infra(4207f3dc-e5c9-4a91-9e71-34b83533046c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.402604 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdj2x\" (UniqueName: \"kubernetes.io/projected/4d8e343e-dbc7-4866-beec-2651bf37a636-kube-api-access-rdj2x\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.402955 5036 scope.go:122] "RemoveContainer" containerID="c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.404799 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume" pod "28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" (UID: "28c4ccea-7e6b-417f-bbf5-fb4caeebdca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.405225 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerDied","Data":"c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed"} Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.407066 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb_odh-ai-gateway-infra(fa478d93-fc7a-4a9b-a3dc-469524500757)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.408281 5036 scope.go:122] "RemoveContainer" containerID="a41ba985c288fff2d1c132d129ce6fb614b9aa8237595b5263b6d035bf20d492" Oct 30 22:00:06.418301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.414301 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp" pod "28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" (UID: "28c4ccea-7e6b-417f-bbf5-fb4caeebdca4"). InnerVolumeSpecName "kube-api-access-vmzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:06.444023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.443813 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 22:00:06.487118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.485761 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-5f75b9c7f7-rqcwg"] Oct 30 22:00:06.493246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.490437 5036 generic.go:356] "Generic (PLEG): container finished" podID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerID="5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951" exitCode=60 Oct 30 22:00:06.493246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.490629 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerDied","Data":"5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951"} Oct 30 22:00:06.493246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.491189 5036 scope.go:122] "RemoveContainer" containerID="5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951" Oct 30 22:00:06.493246 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.491901 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2_odh-ai-gateway-infra(1f8e006c-38db-4d8f-abe6-dbc14851195b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" Oct 30 22:00:06.504384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.503984 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmzbp\" (UniqueName: \"kubernetes.io/projected/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-kube-api-access-vmzbp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:06.504384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.504006 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:06.505689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.505663 5036 scope.go:122] "RemoveContainer" containerID="d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4" Oct 30 22:00:06.506047 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.506019 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wznq9_odh-ai-gateway-infra(8a815ffc-c527-4219-a8dc-518b21c2f4cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" Oct 30 22:00:06.507386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.507337 5036 generic.go:356] "Generic (PLEG): container finished" podID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" containerID="69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1" exitCode=0 Oct 30 22:00:06.514420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.509463 5036 scope.go:122] "RemoveContainer" containerID="3b8cdd297b8b59e3e023e7a2e4ceacd82b6d45c16d5524e27576e81cb9192f93" Oct 30 22:00:06.514420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.509641 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" Oct 30 22:00:06.515432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.510043 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" event={"ID":"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4","Type":"ContainerDied","Data":"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1"} Oct 30 22:00:06.515523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.515474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh" event={"ID":"28c4ccea-7e6b-417f-bbf5-fb4caeebdca4","Type":"ContainerDied","Data":"2436625463a583b81b36d03a2d280dbfdd40e87d3bc603c94579461aeb5149d4"} Oct 30 22:00:06.518188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.518073 5036 scope.go:122] "RemoveContainer" containerID="4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a" Oct 30 22:00:06.518355 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.518333 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf_odh-ai-gateway-infra(8c1036b5-a293-4661-b033-7eab36c87224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" podUID="8c1036b5-a293-4661-b033-7eab36c87224" Oct 30 22:00:06.524020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.524001 5036 generic.go:356] "Generic (PLEG): container finished" podID="f284e515-5326-487f-8c53-f7c87506bae2" containerID="920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428" exitCode=60 Oct 30 22:00:06.524182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.524168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerDied","Data":"920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428"} Oct 30 22:00:06.524986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.524581 5036 scope.go:122] "RemoveContainer" containerID="920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428" Oct 30 22:00:06.524986 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.524929 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq_odh-ai-gateway-infra(f284e515-5326-487f-8c53-f7c87506bae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" podUID="f284e515-5326-487f-8c53-f7c87506bae2" Oct 30 22:00:06.528152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.527581 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fcc2179-6acc-4de1-9573-75a113690029" containerID="ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e" exitCode=60 Oct 30 22:00:06.528152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.527616 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerDied","Data":"ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e"} Oct 30 22:00:06.528152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.527897 5036 scope.go:122] "RemoveContainer" containerID="ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e" Oct 30 22:00:06.528152 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.528115 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf_odh-ai-gateway-infra(2fcc2179-6acc-4de1-9573-75a113690029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" podUID="2fcc2179-6acc-4de1-9573-75a113690029" Oct 30 22:00:06.538179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.538157 5036 scope.go:122] "RemoveContainer" containerID="69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1" Oct 30 22:00:06.552132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.552099 5036 scope.go:122] "RemoveContainer" containerID="69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1" Oct 30 22:00:06.552502 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:06.552470 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1\": container with ID starting with 69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1 not found: ID does not exist" containerID="69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1" Oct 30 22:00:06.552632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.552511 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1"} err="failed to get container status \"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1\": rpc error: code = NotFound desc = could not find container \"69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1\": container with ID starting with 69424c5a20d96f021572851adb5cd8234101121dc005c2f576647ade62fde3a1 not found: ID does not exist" Oct 30 22:00:06.552632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.552538 5036 scope.go:122] "RemoveContainer" containerID="32e972f82292cc402b9cb7202bee0c40e7e6132e88109b541b6920736476ae09" Oct 30 22:00:06.569788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.569739 5036 scope.go:122] "RemoveContainer" containerID="c35d3fcd6a9ce2b2fb3fa24dcd429112f6c57949ec32449545d1967bbb779d68" Oct 30 22:00:06.687691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.687655 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 22:00:06.697601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:06.697572 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-7678dfd887-4q5kh"] Oct 30 22:00:07.542534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:07.542508 5036 scope.go:122] "RemoveContainer" containerID="5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951" Oct 30 22:00:07.543017 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:07.542921 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2_odh-ai-gateway-infra(1f8e006c-38db-4d8f-abe6-dbc14851195b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" Oct 30 22:00:07.546189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:07.546164 5036 scope.go:122] "RemoveContainer" containerID="920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428" Oct 30 22:00:07.546592 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:07.546548 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq_odh-ai-gateway-infra(f284e515-5326-487f-8c53-f7c87506bae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" podUID="f284e515-5326-487f-8c53-f7c87506bae2" Oct 30 22:00:07.548677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:07.548653 5036 scope.go:122] "RemoveContainer" containerID="ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e" Oct 30 22:00:07.549007 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:07.548980 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf_odh-ai-gateway-infra(2fcc2179-6acc-4de1-9573-75a113690029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" podUID="2fcc2179-6acc-4de1-9573-75a113690029" Oct 30 22:00:07.551813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:07.551796 5036 scope.go:122] "RemoveContainer" containerID="f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477" Oct 30 22:00:07.552103 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:07.552083 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw_odh-ai-gateway-infra(c5d9a1be-eddc-4de6-b600-1407aada133b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" Oct 30 22:00:07.553808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:07.553786 5036 scope.go:122] "RemoveContainer" containerID="c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed" Oct 30 22:00:07.554008 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:07.553992 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb_odh-ai-gateway-infra(fa478d93-fc7a-4a9b-a3dc-469524500757)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" Oct 30 22:00:08.307353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.303981 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" path="/var/lib/kubelet/pods/28c4ccea-7e6b-417f-bbf5-fb4caeebdca4/volumes" Oct 30 22:00:08.307353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.304739 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8e343e-dbc7-4866-beec-2651bf37a636" path="/var/lib/kubelet/pods/4d8e343e-dbc7-4866-beec-2651bf37a636/volumes" Oct 30 22:00:08.440348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.434422 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5"] Oct 30 22:00:08.441177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.441151 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f47f79-59tk5"] Oct 30 22:00:08.575796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.575684 5036 generic.go:356] "Generic (PLEG): container finished" podID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerID="eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f" exitCode=0 Oct 30 22:00:08.575796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.575763 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerDied","Data":"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f"} Oct 30 22:00:08.576407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.576373 5036 scope.go:122] "RemoveContainer" containerID="eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f" Oct 30 22:00:08.578963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.578924 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:08.579088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.579042 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:08.579147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:08.579099 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:09.461324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:09.460454 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:09.713345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:09.697719 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerStarted","Data":"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f"} Oct 30 22:00:09.781292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:09.777496 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr"] Oct 30 22:00:09.989856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:09.989774 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d2c42-8vnjr"] Oct 30 22:00:10.071299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.069786 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:10.073557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.071997 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\") pod \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " Oct 30 22:00:10.073557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.072107 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\" (UniqueName: \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\") pod \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\" (UID: \"ee085cb9-48f1-4cc3-bc3d-b5649bf69064\") " Oct 30 22:00:10.073557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.073039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca" pod "ee085cb9-48f1-4cc3-bc3d-b5649bf69064" (UID: "ee085cb9-48f1-4cc3-bc3d-b5649bf69064"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:10.075806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.075780 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg" pod "ee085cb9-48f1-4cc3-bc3d-b5649bf69064" (UID: "ee085cb9-48f1-4cc3-bc3d-b5649bf69064"). InnerVolumeSpecName "kube-api-access-kh5zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:10.090192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.090087 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:10.090497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.090466 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:10.090626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.090536 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:10.216825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.199826 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:10.216825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.199868 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh5zg\" (UniqueName: \"kubernetes.io/projected/ee085cb9-48f1-4cc3-bc3d-b5649bf69064-kube-api-access-kh5zg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:10.277878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.277778 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3836a17c-9eeb-4bd0-bd17-8f77fc66e340" path="/var/lib/kubelet/pods/3836a17c-9eeb-4bd0-bd17-8f77fc66e340/volumes" Oct 30 22:00:10.279067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.278449 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399df661-1f8e-4bde-8d63-5128243aacda" path="/var/lib/kubelet/pods/399df661-1f8e-4bde-8d63-5128243aacda/volumes" Oct 30 22:00:10.710669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.710639 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" Oct 30 22:00:10.716283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.715403 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484" event={"ID":"ee085cb9-48f1-4cc3-bc3d-b5649bf69064","Type":"ContainerDied","Data":"186d7a6dc069cb260b5e5e7d90edb730bb001f8a613031b930c37941a58a53e7"} Oct 30 22:00:10.716283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.715476 5036 scope.go:122] "RemoveContainer" containerID="8aab74e4128abb85b6119b03fd49b1d3c12dbc817dafcb2c8c5fc8750c3ec699" Oct 30 22:00:10.772568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.772537 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:10.782628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:10.782600 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d2c42-29889960-hh484"] Oct 30 22:00:11.091417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:11.091321 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:11.091417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:11.091376 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:11.244853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:11.241371 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb"] Oct 30 22:00:11.306218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:11.306179 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-208470cf-tg5gb"] Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.090709 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092060 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092083 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092137 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092146 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092178 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4d8e343e-dbc7-4866-beec-2651bf37a636" containerName="payload-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092187 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8e343e-dbc7-4866-beec-2651bf37a636" containerName="payload-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092207 5036 cpu_manager.go:414] "containerMap: removing container" podUID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" containerName="payload-pre-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092215 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" containerName="payload-pre-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092444 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8e343e-dbc7-4866-beec-2651bf37a636" containerName="payload-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092465 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c4ccea-7e6b-417f-bbf5-fb4caeebdca4" containerName="payload-pre-processing" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092488 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092507 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" containerName="cleanup" Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092580 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:12.094287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.092627 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:12.174294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.173580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:00:12.174294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.173637 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:00:12.188328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.182745 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.190411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.189558 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-19cab1\"" Oct 30 22:00:12.301313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.284446 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.301313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.294777 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-19cab1-dockercfg-9hg27\"" Oct 30 22:00:12.453298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.448120 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:00:12.493359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.491277 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.493359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.491341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j7t\" (UniqueName: \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.493359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.491400 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.493359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.491594 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7w4\" (UniqueName: \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.531293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.511160 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7fefd0-b8d8-4ab3-8610-a45d61897334" path="/var/lib/kubelet/pods/ab7fefd0-b8d8-4ab3-8610-a45d61897334/volumes" Oct 30 22:00:12.531293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.511884 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee085cb9-48f1-4cc3-bc3d-b5649bf69064" path="/var/lib/kubelet/pods/ee085cb9-48f1-4cc3-bc3d-b5649bf69064/volumes" Oct 30 22:00:12.594184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.594148 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7w4\" (UniqueName: \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.594667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.594630 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.594879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.594859 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j7t\" (UniqueName: \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.594953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.594917 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.596231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.595538 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.596231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.596112 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.634326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.616096 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j7t\" (UniqueName: \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\") pod \"payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.656327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.654020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7w4\" (UniqueName: \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\") pod \"payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:12.833346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.808742 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:12.833346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:12.809564 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:13.089979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.089901 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:13.089979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.089958 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:13.209009 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:13.208972 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58b9493_f7d4_4cee_b938_b1192996e5a0.slice/crio-009759dd462e494b436409843ddf189a30cd7679eda92d13777a25b3d7f93300 WatchSource:0}: Error finding container 009759dd462e494b436409843ddf189a30cd7679eda92d13777a25b3d7f93300: Status 404 returned error can't find the container with id 009759dd462e494b436409843ddf189a30cd7679eda92d13777a25b3d7f93300 Oct 30 22:00:13.209413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.209391 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:00:13.253302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.251571 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.811329 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" event={"ID":"b58b9493-f7d4-4cee-b938-b1192996e5a0","Type":"ContainerStarted","Data":"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c"} Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.811398 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" event={"ID":"b58b9493-f7d4-4cee-b938-b1192996e5a0","Type":"ContainerStarted","Data":"009759dd462e494b436409843ddf189a30cd7679eda92d13777a25b3d7f93300"} Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.812699 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.815641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" event={"ID":"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4","Type":"ContainerStarted","Data":"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170"} Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.815690 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" event={"ID":"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4","Type":"ContainerStarted","Data":"bfe1e3352e47ad1989fb6dee2f392a5612b0dadd1777f58ebe3cc52fa79afdc8"} Oct 30 22:00:13.820310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.816479 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:13.865285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.862564 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" podStartSLOduration=1.862548874 podStartE2EDuration="1.862548874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:13.843634704 +0000 UTC m=+1237.793589650" watchObservedRunningTime="2026-10-30 22:00:13.862548874 +0000 UTC m=+1237.812503839" Oct 30 22:00:13.880963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:13.880919 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" podStartSLOduration=1.8808974630000002 podStartE2EDuration="1.880897463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:13.877421411 +0000 UTC m=+1237.827376358" watchObservedRunningTime="2026-10-30 22:00:13.880897463 +0000 UTC m=+1237.830852411" Oct 30 22:00:14.102901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:14.102810 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:14.102901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:14.102871 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:15.091229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:15.089970 5036 patch_prober.go:29] interesting pod/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:00:15.091229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:15.091075 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:00:15.556975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:15.556929 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:16.095986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:16.095959 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:16.097378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:16.097355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:16.188529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:16.188365 5036 scope.go:122] "RemoveContainer" containerID="ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee" Oct 30 22:00:17.181475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:17.180281 5036 scope.go:122] "RemoveContainer" containerID="c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed" Oct 30 22:00:17.184347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:17.182151 5036 scope.go:122] "RemoveContainer" containerID="4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a" Oct 30 22:00:17.184347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:17.182966 5036 scope.go:122] "RemoveContainer" containerID="f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477" Oct 30 22:00:18.066892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.066858 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerStarted","Data":"cac2ae359202cebb598bf96f461d8b41c1387f91f34a05644d8037361516f6dc"} Oct 30 22:00:18.069345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.069229 5036 generic.go:356] "Generic (PLEG): container finished" podID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerID="e0df920f5c2302812728b1fd4f55e9768f1e9f55f1dd9a484a0722f29ba5d31f" exitCode=60 Oct 30 22:00:18.069345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.069298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerDied","Data":"e0df920f5c2302812728b1fd4f55e9768f1e9f55f1dd9a484a0722f29ba5d31f"} Oct 30 22:00:18.069545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.069351 5036 scope.go:122] "RemoveContainer" containerID="ed90ea345380c768a6c6fda8206fe35b52b0316f445e65a85473284d341fbdee" Oct 30 22:00:18.069819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.069698 5036 scope.go:122] "RemoveContainer" containerID="e0df920f5c2302812728b1fd4f55e9768f1e9f55f1dd9a484a0722f29ba5d31f" Oct 30 22:00:18.070038 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:18.070019 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz_odh-ai-gateway-infra(4207f3dc-e5c9-4a91-9e71-34b83533046c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" Oct 30 22:00:18.073025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.072361 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerStarted","Data":"d4d84457d50c25d5bf96fad976c5f1e260a4c4960cb6f9e1320c5bccacdc68f5"} Oct 30 22:00:18.087984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.087958 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerStarted","Data":"b64ffd876fc08f9681579497218d54a09eac7ac7ef12bb53ba63bdb8922b4f53"} Oct 30 22:00:18.102053 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:18.102020 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:00:18.180975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.180942 5036 scope.go:122] "RemoveContainer" containerID="5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951" Oct 30 22:00:18.181177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.181155 5036 scope.go:122] "RemoveContainer" containerID="ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e" Oct 30 22:00:18.961182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.961153 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 22:00:18.961601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:18.961356 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" containerID="cri-o://30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f" gracePeriod=30 Oct 30 22:00:19.172342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.171897 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:19.202434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.189138 5036 scope.go:122] "RemoveContainer" containerID="d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4" Oct 30 22:00:19.202434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.190302 5036 scope.go:122] "RemoveContainer" containerID="920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428" Oct 30 22:00:19.235554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.232544 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:19.248298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.248241 5036 generic.go:356] "Generic (PLEG): container finished" podID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerID="d4d84457d50c25d5bf96fad976c5f1e260a4c4960cb6f9e1320c5bccacdc68f5" exitCode=60 Oct 30 22:00:19.248507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.248457 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerDied","Data":"d4d84457d50c25d5bf96fad976c5f1e260a4c4960cb6f9e1320c5bccacdc68f5"} Oct 30 22:00:19.248594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.248518 5036 scope.go:122] "RemoveContainer" containerID="c43434e60a1870da99f50f09a243eb4494e2a5fcf25dbd2416ef2a5da60083ed" Oct 30 22:00:19.251367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.249019 5036 scope.go:122] "RemoveContainer" containerID="d4d84457d50c25d5bf96fad976c5f1e260a4c4960cb6f9e1320c5bccacdc68f5" Oct 30 22:00:19.251367 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:19.250613 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb_odh-ai-gateway-infra(fa478d93-fc7a-4a9b-a3dc-469524500757)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" Oct 30 22:00:19.267540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.266339 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerStarted","Data":"8bd59b966f2e7ce2e590fe3395fdc99f2bef9ce184cd024fe45fb9aa99e640b5"} Oct 30 22:00:19.301544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.289613 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c1036b5-a293-4661-b033-7eab36c87224" containerID="b64ffd876fc08f9681579497218d54a09eac7ac7ef12bb53ba63bdb8922b4f53" exitCode=60 Oct 30 22:00:19.301544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.289779 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerDied","Data":"b64ffd876fc08f9681579497218d54a09eac7ac7ef12bb53ba63bdb8922b4f53"} Oct 30 22:00:19.302921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.302083 5036 scope.go:122] "RemoveContainer" containerID="b64ffd876fc08f9681579497218d54a09eac7ac7ef12bb53ba63bdb8922b4f53" Oct 30 22:00:19.302921 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:19.302578 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf_odh-ai-gateway-infra(8c1036b5-a293-4661-b033-7eab36c87224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" podUID="8c1036b5-a293-4661-b033-7eab36c87224" Oct 30 22:00:19.379292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.366804 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerStarted","Data":"e3962684984c8c84a4b17759f9def6e7524812b5ec51c446bd96e7cec8f12586"} Oct 30 22:00:19.379292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.370490 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerID="cac2ae359202cebb598bf96f461d8b41c1387f91f34a05644d8037361516f6dc" exitCode=60 Oct 30 22:00:19.379292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.370574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerDied","Data":"cac2ae359202cebb598bf96f461d8b41c1387f91f34a05644d8037361516f6dc"} Oct 30 22:00:19.405005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.404950 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:19.406828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.406711 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:19.412709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.410503 5036 scope.go:122] "RemoveContainer" containerID="4e6f3f9a644826e2fee18a900c412106e5f7d5b4c2ef59964c5301112ed6700a" Oct 30 22:00:19.517297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.514246 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw"] Oct 30 22:00:19.530381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.529937 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.536443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.536367 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw"] Oct 30 22:00:19.540303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.540131 5036 scope.go:122] "RemoveContainer" containerID="f1c5279767b456440c0a3443aedff3c9bf9c84e01689bb04dbeb1e3ceb857477" Oct 30 22:00:19.599594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.599565 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:00:19.641654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.641623 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.641923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.641898 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.642426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.642405 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbb57\" (UniqueName: \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.725975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.725915 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:19.731720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.731689 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:19.744049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.744023 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.744259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.744223 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.744515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.744494 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbb57\" (UniqueName: \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.745186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.745160 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.757017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.753660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.757822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.757802 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbb57\" (UniqueName: \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\") pod \"maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.846175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\" (UniqueName: \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\") pod \"4207f3dc-e5c9-4a91-9e71-34b83533046c\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " Oct 30 22:00:19.846391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846373 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\") pod \"c5d9a1be-eddc-4de6-b600-1407aada133b\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " Oct 30 22:00:19.846483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\" (UniqueName: \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\") pod \"c5d9a1be-eddc-4de6-b600-1407aada133b\" (UID: \"c5d9a1be-eddc-4de6-b600-1407aada133b\") " Oct 30 22:00:19.846555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\") pod \"4207f3dc-e5c9-4a91-9e71-34b83533046c\" (UID: \"4207f3dc-e5c9-4a91-9e71-34b83533046c\") " Oct 30 22:00:19.846872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846840 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca" pod "c5d9a1be-eddc-4de6-b600-1407aada133b" (UID: "c5d9a1be-eddc-4de6-b600-1407aada133b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.847021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.846998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca" pod "4207f3dc-e5c9-4a91-9e71-34b83533046c" (UID: "4207f3dc-e5c9-4a91-9e71-34b83533046c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.847506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.847486 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d9a1be-eddc-4de6-b600-1407aada133b-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.847596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.847510 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4207f3dc-e5c9-4a91-9e71-34b83533046c-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.848511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.848492 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf" pod "4207f3dc-e5c9-4a91-9e71-34b83533046c" (UID: "4207f3dc-e5c9-4a91-9e71-34b83533046c"). InnerVolumeSpecName "kube-api-access-hmmhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.848900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.848876 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w" pod "c5d9a1be-eddc-4de6-b600-1407aada133b" (UID: "c5d9a1be-eddc-4de6-b600-1407aada133b"). InnerVolumeSpecName "kube-api-access-vmw8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.887627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.887603 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:19.953444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.953397 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmw8w\" (UniqueName: \"kubernetes.io/projected/c5d9a1be-eddc-4de6-b600-1407aada133b-kube-api-access-vmw8w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.954242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:19.954182 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmhf\" (UniqueName: \"kubernetes.io/projected/4207f3dc-e5c9-4a91-9e71-34b83533046c-kube-api-access-hmmhf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.412711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.412198 5036 generic.go:356] "Generic (PLEG): container finished" podID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerID="8bd59b966f2e7ce2e590fe3395fdc99f2bef9ce184cd024fe45fb9aa99e640b5" exitCode=60 Oct 30 22:00:20.417677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.417639 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw"] Oct 30 22:00:20.417855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.417831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerDied","Data":"8bd59b966f2e7ce2e590fe3395fdc99f2bef9ce184cd024fe45fb9aa99e640b5"} Oct 30 22:00:20.417941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.417875 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" event={"ID":"94f0c3cf-1c83-440a-a8e0-9288ae2b0672","Type":"ContainerStarted","Data":"4a90e0db00a766f7cd4989c3b59b0a746fe75ad2017ff57dd1c0002e87796377"} Oct 30 22:00:20.417941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.417907 5036 scope.go:122] "RemoveContainer" containerID="5ca5e508d836009361206016ab94661ee9a6c4054443d67c7fdffc36a8e09951" Oct 30 22:00:20.425812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.425790 5036 scope.go:122] "RemoveContainer" containerID="8bd59b966f2e7ce2e590fe3395fdc99f2bef9ce184cd024fe45fb9aa99e640b5" Oct 30 22:00:20.426193 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:20.426165 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2_odh-ai-gateway-infra(1f8e006c-38db-4d8f-abe6-dbc14851195b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" Oct 30 22:00:20.431517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.431492 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerStarted","Data":"e4f4f7d3c16c8efa3adb5e215dd3add352004dba62f1e70b7472ec66d9a83be5"} Oct 30 22:00:20.433933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.433907 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fcc2179-6acc-4de1-9573-75a113690029" containerID="e3962684984c8c84a4b17759f9def6e7524812b5ec51c446bd96e7cec8f12586" exitCode=60 Oct 30 22:00:20.434060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.434045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerDied","Data":"e3962684984c8c84a4b17759f9def6e7524812b5ec51c446bd96e7cec8f12586"} Oct 30 22:00:20.434426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.434409 5036 scope.go:122] "RemoveContainer" containerID="e3962684984c8c84a4b17759f9def6e7524812b5ec51c446bd96e7cec8f12586" Oct 30 22:00:20.434695 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:20.434672 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf_odh-ai-gateway-infra(2fcc2179-6acc-4de1-9573-75a113690029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" podUID="2fcc2179-6acc-4de1-9573-75a113690029" Oct 30 22:00:20.439681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.437396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" event={"ID":"c5d9a1be-eddc-4de6-b600-1407aada133b","Type":"ContainerDied","Data":"4dc0beebd3f3bfc51e895d50822d34bdf30e7dcc70342caf4fb32064d54e497b"} Oct 30 22:00:20.439681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.437495 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw" Oct 30 22:00:20.440255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.440221 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:20.443981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.443955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" event={"ID":"4207f3dc-e5c9-4a91-9e71-34b83533046c","Type":"ContainerDied","Data":"019627e378f6147ae230e6c7b0bfc21bbcb4edf6aebc497246740075f5dfc4d4"} Oct 30 22:00:20.444077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.444060 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz" Oct 30 22:00:20.479333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.479302 5036 scope.go:122] "RemoveContainer" containerID="ce257ad1ec51d435d9cfe05f3bbf265cd84c0030c58fa95c73ebb1742d88061e" Oct 30 22:00:20.482369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.481515 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:20.503535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.503505 5036 scope.go:122] "RemoveContainer" containerID="cac2ae359202cebb598bf96f461d8b41c1387f91f34a05644d8037361516f6dc" Oct 30 22:00:20.521359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.521332 5036 scope.go:122] "RemoveContainer" containerID="e0df920f5c2302812728b1fd4f55e9768f1e9f55f1dd9a484a0722f29ba5d31f" Oct 30 22:00:20.659398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.659368 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:20.665668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.665639 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889960-flrvw"] Oct 30 22:00:20.675617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.675588 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:20.682883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.682856 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c614f9-29889960-cdrpz"] Oct 30 22:00:20.877658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.877626 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:20.891142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.891091 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:20.962167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.962130 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\" (UniqueName: \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\") pod \"8c1036b5-a293-4661-b033-7eab36c87224\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " Oct 30 22:00:20.962167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.962175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\") pod \"8c1036b5-a293-4661-b033-7eab36c87224\" (UID: \"8c1036b5-a293-4661-b033-7eab36c87224\") " Oct 30 22:00:20.962475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.962457 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\") pod \"fa478d93-fc7a-4a9b-a3dc-469524500757\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " Oct 30 22:00:20.962539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.962491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\" (UniqueName: \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\") pod \"fa478d93-fc7a-4a9b-a3dc-469524500757\" (UID: \"fa478d93-fc7a-4a9b-a3dc-469524500757\") " Oct 30 22:00:20.962627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.962604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca" pod "8c1036b5-a293-4661-b033-7eab36c87224" (UID: "8c1036b5-a293-4661-b033-7eab36c87224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.963297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.963229 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1036b5-a293-4661-b033-7eab36c87224-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.963431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.963342 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca" pod "fa478d93-fc7a-4a9b-a3dc-469524500757" (UID: "fa478d93-fc7a-4a9b-a3dc-469524500757"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.964967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.964929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4" pod "fa478d93-fc7a-4a9b-a3dc-469524500757" (UID: "fa478d93-fc7a-4a9b-a3dc-469524500757"). InnerVolumeSpecName "kube-api-access-ndwm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.965134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:20.965106 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl" pod "8c1036b5-a293-4661-b033-7eab36c87224" (UID: "8c1036b5-a293-4661-b033-7eab36c87224"). InnerVolumeSpecName "kube-api-access-jc6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.067169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.067130 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa478d93-fc7a-4a9b-a3dc-469524500757-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.067444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.067174 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwm4\" (UniqueName: \"kubernetes.io/projected/fa478d93-fc7a-4a9b-a3dc-469524500757-kube-api-access-ndwm4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.067444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.067195 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6gl\" (UniqueName: \"kubernetes.io/projected/8c1036b5-a293-4661-b033-7eab36c87224-kube-api-access-jc6gl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.461932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.461899 5036 generic.go:356] "Generic (PLEG): container finished" podID="f284e515-5326-487f-8c53-f7c87506bae2" containerID="e4f4f7d3c16c8efa3adb5e215dd3add352004dba62f1e70b7472ec66d9a83be5" exitCode=60 Oct 30 22:00:21.462382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.461968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerDied","Data":"e4f4f7d3c16c8efa3adb5e215dd3add352004dba62f1e70b7472ec66d9a83be5"} Oct 30 22:00:21.462382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.462044 5036 scope.go:122] "RemoveContainer" containerID="920095649b500b08bb566d1796612c1b28a553d79e486a8d86c2567d3e869428" Oct 30 22:00:21.462576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.462557 5036 scope.go:122] "RemoveContainer" containerID="e4f4f7d3c16c8efa3adb5e215dd3add352004dba62f1e70b7472ec66d9a83be5" Oct 30 22:00:21.463033 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:21.463002 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq_odh-ai-gateway-infra(f284e515-5326-487f-8c53-f7c87506bae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" podUID="f284e515-5326-487f-8c53-f7c87506bae2" Oct 30 22:00:21.468726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.468706 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" Oct 30 22:00:21.468726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.468713 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb" event={"ID":"fa478d93-fc7a-4a9b-a3dc-469524500757","Type":"ContainerDied","Data":"91bfa0b87cda7b84efd3a6ad9f8504f59cb918a743c822b8236b684c3ccf788c"} Oct 30 22:00:21.476324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.476295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerDied","Data":"695eaef60aa364dc66c89ab32ab0c6cf98187891f4511db47a714b29a8f347fa"} Oct 30 22:00:21.476443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.476308 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerID="695eaef60aa364dc66c89ab32ab0c6cf98187891f4511db47a714b29a8f347fa" exitCode=60 Oct 30 22:00:21.476770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.476750 5036 scope.go:122] "RemoveContainer" containerID="695eaef60aa364dc66c89ab32ab0c6cf98187891f4511db47a714b29a8f347fa" Oct 30 22:00:21.477095 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:21.477071 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wznq9_odh-ai-gateway-infra(8a815ffc-c527-4219-a8dc-518b21c2f4cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" Oct 30 22:00:21.478648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.478604 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" event={"ID":"94f0c3cf-1c83-440a-a8e0-9288ae2b0672","Type":"ContainerStarted","Data":"6b7bb52ca635f2e0c1f6dff52f70a6036b654bb0ee16c0607416d1f3c988b513"} Oct 30 22:00:21.498987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.498827 5036 scope.go:122] "RemoveContainer" containerID="d4d84457d50c25d5bf96fad976c5f1e260a4c4960cb6f9e1320c5bccacdc68f5" Oct 30 22:00:21.542281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.533706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" event={"ID":"8c1036b5-a293-4661-b033-7eab36c87224","Type":"ContainerDied","Data":"8d658244ce682ebca31601b50757d239bdb5aa2e3a5a047a79dfcd38411cb3d9"} Oct 30 22:00:21.542281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.533854 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf" Oct 30 22:00:21.597845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.597221 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" podStartSLOduration=2.5972024449999997 podStartE2EDuration="2.597202445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:21.548786307 +0000 UTC m=+1245.498741255" watchObservedRunningTime="2026-10-30 22:00:21.597202445 +0000 UTC m=+1245.547157392" Oct 30 22:00:21.618570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.606718 5036 scope.go:122] "RemoveContainer" containerID="d1a52d5cff913b179cb4141df1e368c7445190766ed145478e174223010b97a4" Oct 30 22:00:21.618570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.617337 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:21.658030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.657755 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c6cf1-29889960-bpsjb"] Oct 30 22:00:21.679418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.676318 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:21.714385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.714311 5036 scope.go:122] "RemoveContainer" containerID="b64ffd876fc08f9681579497218d54a09eac7ac7ef12bb53ba63bdb8922b4f53" Oct 30 22:00:21.728062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.727993 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:21.745669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.745188 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3654d5-29889960-8pbjf"] Oct 30 22:00:21.841834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.841807 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:21.843846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.843831 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:21.885631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.885596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\") pod \"1f8e006c-38db-4d8f-abe6-dbc14851195b\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " Oct 30 22:00:21.885833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.885773 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\" (UniqueName: \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\") pod \"1f8e006c-38db-4d8f-abe6-dbc14851195b\" (UID: \"1f8e006c-38db-4d8f-abe6-dbc14851195b\") " Oct 30 22:00:21.885917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.885840 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\" (UniqueName: \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\") pod \"2fcc2179-6acc-4de1-9573-75a113690029\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " Oct 30 22:00:21.885917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.885891 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\") pod \"2fcc2179-6acc-4de1-9573-75a113690029\" (UID: \"2fcc2179-6acc-4de1-9573-75a113690029\") " Oct 30 22:00:21.887153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.887121 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca" pod "1f8e006c-38db-4d8f-abe6-dbc14851195b" (UID: "1f8e006c-38db-4d8f-abe6-dbc14851195b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.887307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.887279 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca" pod "2fcc2179-6acc-4de1-9573-75a113690029" (UID: "2fcc2179-6acc-4de1-9573-75a113690029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.889354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.889246 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw" pod "2fcc2179-6acc-4de1-9573-75a113690029" (UID: "2fcc2179-6acc-4de1-9573-75a113690029"). InnerVolumeSpecName "kube-api-access-k7pqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.889723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.889706 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz" pod "1f8e006c-38db-4d8f-abe6-dbc14851195b" (UID: "1f8e006c-38db-4d8f-abe6-dbc14851195b"). InnerVolumeSpecName "kube-api-access-27nfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.988519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.988427 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27nfz\" (UniqueName: \"kubernetes.io/projected/1f8e006c-38db-4d8f-abe6-dbc14851195b-kube-api-access-27nfz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.988519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.988459 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7pqw\" (UniqueName: \"kubernetes.io/projected/2fcc2179-6acc-4de1-9573-75a113690029-kube-api-access-k7pqw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.988519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.988472 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc2179-6acc-4de1-9573-75a113690029-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.988519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:21.988482 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8e006c-38db-4d8f-abe6-dbc14851195b-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:22.208240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.208205 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" path="/var/lib/kubelet/pods/4207f3dc-e5c9-4a91-9e71-34b83533046c/volumes" Oct 30 22:00:22.208862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.208838 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1036b5-a293-4661-b033-7eab36c87224" path="/var/lib/kubelet/pods/8c1036b5-a293-4661-b033-7eab36c87224/volumes" Oct 30 22:00:22.209379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.209356 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" path="/var/lib/kubelet/pods/c5d9a1be-eddc-4de6-b600-1407aada133b/volumes" Oct 30 22:00:22.210170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.210146 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" path="/var/lib/kubelet/pods/fa478d93-fc7a-4a9b-a3dc-469524500757/volumes" Oct 30 22:00:22.536094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.536002 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:22.548885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.548857 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" Oct 30 22:00:22.549141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.548857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf" event={"ID":"2fcc2179-6acc-4de1-9573-75a113690029","Type":"ContainerDied","Data":"3cfeaf4bcf20252bd0928ff2501717a7996118ef3348a0e9a3eb03b5ee3ba51d"} Oct 30 22:00:22.549141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.549009 5036 scope.go:122] "RemoveContainer" containerID="e3962684984c8c84a4b17759f9def6e7524812b5ec51c446bd96e7cec8f12586" Oct 30 22:00:22.554725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.554456 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" event={"ID":"1f8e006c-38db-4d8f-abe6-dbc14851195b","Type":"ContainerDied","Data":"994b5a7c0e7213c147a4606f854a2c4568c71deca97314272bbc64e328ad4bdb"} Oct 30 22:00:22.554725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.554542 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2" Oct 30 22:00:22.560563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.560533 5036 generic.go:356] "Generic (PLEG): container finished" podID="94f0c3cf-1c83-440a-a8e0-9288ae2b0672" containerID="6b7bb52ca635f2e0c1f6dff52f70a6036b654bb0ee16c0607416d1f3c988b513" exitCode=0 Oct 30 22:00:22.560678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.560614 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" event={"ID":"94f0c3cf-1c83-440a-a8e0-9288ae2b0672","Type":"ContainerDied","Data":"6b7bb52ca635f2e0c1f6dff52f70a6036b654bb0ee16c0607416d1f3c988b513"} Oct 30 22:00:22.567716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.567694 5036 scope.go:122] "RemoveContainer" containerID="8bd59b966f2e7ce2e590fe3395fdc99f2bef9ce184cd024fe45fb9aa99e640b5" Oct 30 22:00:22.590142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.590110 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:22.598671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.598640 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3670-29889960-q4hl2"] Oct 30 22:00:22.647121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.645638 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:22.657043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.657016 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889960-j9szf"] Oct 30 22:00:22.790877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.790821 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:22.799574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.799552 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\") pod \"f284e515-5326-487f-8c53-f7c87506bae2\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " Oct 30 22:00:22.799742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.799726 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\" (UniqueName: \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\") pod \"f284e515-5326-487f-8c53-f7c87506bae2\" (UID: \"f284e515-5326-487f-8c53-f7c87506bae2\") " Oct 30 22:00:22.799882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.799861 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca" pod "f284e515-5326-487f-8c53-f7c87506bae2" (UID: "f284e515-5326-487f-8c53-f7c87506bae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:22.800309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.800259 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284e515-5326-487f-8c53-f7c87506bae2-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:22.802138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.802115 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr" pod "f284e515-5326-487f-8c53-f7c87506bae2" (UID: "f284e515-5326-487f-8c53-f7c87506bae2"). InnerVolumeSpecName "kube-api-access-z4blr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:22.833257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.833231 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:22.901535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.901504 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\" (UniqueName: \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\") pod \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " Oct 30 22:00:22.901710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.901645 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\") pod \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\" (UID: \"8a815ffc-c527-4219-a8dc-518b21c2f4cc\") " Oct 30 22:00:22.902252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.902224 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4blr\" (UniqueName: \"kubernetes.io/projected/f284e515-5326-487f-8c53-f7c87506bae2-kube-api-access-z4blr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:22.902412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.902254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca" pod "8a815ffc-c527-4219-a8dc-518b21c2f4cc" (UID: "8a815ffc-c527-4219-a8dc-518b21c2f4cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:22.904557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:22.904525 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87" pod "8a815ffc-c527-4219-a8dc-518b21c2f4cc" (UID: "8a815ffc-c527-4219-a8dc-518b21c2f4cc"). InnerVolumeSpecName "kube-api-access-v9m87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:23.003458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.003425 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a815ffc-c527-4219-a8dc-518b21c2f4cc-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.003458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.003452 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9m87\" (UniqueName: \"kubernetes.io/projected/8a815ffc-c527-4219-a8dc-518b21c2f4cc-kube-api-access-v9m87\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.579918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.579878 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" event={"ID":"f284e515-5326-487f-8c53-f7c87506bae2","Type":"ContainerDied","Data":"a522e3a24186ae241488e49d524a653c0358c6d7cca6c583c329eea4f5cbc986"} Oct 30 22:00:23.580489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.579950 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq" Oct 30 22:00:23.580489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.579965 5036 scope.go:122] "RemoveContainer" containerID="e4f4f7d3c16c8efa3adb5e215dd3add352004dba62f1e70b7472ec66d9a83be5" Oct 30 22:00:23.594705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.594671 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" event={"ID":"8a815ffc-c527-4219-a8dc-518b21c2f4cc","Type":"ContainerDied","Data":"cc76ecd221af6103ef5fdddfd9b4edfe05c695f31c26a1d0e7a5c096d6f0de60"} Oct 30 22:00:23.594819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.594685 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9" Oct 30 22:00:23.603727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.603684 5036 scope.go:122] "RemoveContainer" containerID="695eaef60aa364dc66c89ab32ab0c6cf98187891f4511db47a714b29a8f347fa" Oct 30 22:00:23.611845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.611821 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:23.620695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.620670 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-19cab1-29889960-fv2kq"] Oct 30 22:00:23.638192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.638061 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:23.655978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.655947 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wznq9"] Oct 30 22:00:23.813727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.813697 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:23.918690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.918656 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\") pod \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " Oct 30 22:00:23.918877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.918860 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\") pod \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " Oct 30 22:00:23.918941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.918912 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\" (UniqueName: \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\") pod \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\" (UID: \"94f0c3cf-1c83-440a-a8e0-9288ae2b0672\") " Oct 30 22:00:23.919281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.919232 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca" pod "94f0c3cf-1c83-440a-a8e0-9288ae2b0672" (UID: "94f0c3cf-1c83-440a-a8e0-9288ae2b0672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:23.919700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.919673 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.921107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.921075 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca" pod "94f0c3cf-1c83-440a-a8e0-9288ae2b0672" (UID: "94f0c3cf-1c83-440a-a8e0-9288ae2b0672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:23.921244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:23.921220 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57" pod "94f0c3cf-1c83-440a-a8e0-9288ae2b0672" (UID: "94f0c3cf-1c83-440a-a8e0-9288ae2b0672"). InnerVolumeSpecName "kube-api-access-hbb57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:24.021547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.021510 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbb57\" (UniqueName: \"kubernetes.io/projected/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-kube-api-access-hbb57\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.021547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.021543 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f0c3cf-1c83-440a-a8e0-9288ae2b0672-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.037900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.037816 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8"] Oct 30 22:00:24.042773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.042748 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d72c72-6hmq8"] Oct 30 22:00:24.193792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.193758 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0779815a-7935-4123-9a31-59d68b2abd67" path="/var/lib/kubelet/pods/0779815a-7935-4123-9a31-59d68b2abd67/volumes" Oct 30 22:00:24.194304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.194253 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" path="/var/lib/kubelet/pods/1f8e006c-38db-4d8f-abe6-dbc14851195b/volumes" Oct 30 22:00:24.194696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.194681 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcc2179-6acc-4de1-9573-75a113690029" path="/var/lib/kubelet/pods/2fcc2179-6acc-4de1-9573-75a113690029/volumes" Oct 30 22:00:24.195427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.195406 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" path="/var/lib/kubelet/pods/8a815ffc-c527-4219-a8dc-518b21c2f4cc/volumes" Oct 30 22:00:24.195856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.195839 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f284e515-5326-487f-8c53-f7c87506bae2" path="/var/lib/kubelet/pods/f284e515-5326-487f-8c53-f7c87506bae2/volumes" Oct 30 22:00:24.305929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.305848 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:24.333525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333487 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333679 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333735 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.333969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333918 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:24.333969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.333960 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.334072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334021 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.334122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\") pod \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\" (UID: \"f45a8d71-db14-47f4-a20e-7f7ebb33d20a\") " Oct 30 22:00:24.334403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334348 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:24.334503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334425 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:24.334626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334602 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:24.334954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.334934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:24.335086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.335069 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.335156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.335093 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.335156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.335107 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.335156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.335121 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.335156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.335135 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.338521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.338242 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "kube-api-access-m6t2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:24.338521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.338399 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:24.338521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.338488 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:24.338873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.338842 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token" pod "f45a8d71-db14-47f4-a20e-7f7ebb33d20a" (UID: "f45a8d71-db14-47f4-a20e-7f7ebb33d20a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:24.436524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.436434 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.436524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.436467 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.436524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.436482 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6t2x\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-kube-api-access-m6t2x\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.436524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.436498 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45a8d71-db14-47f4-a20e-7f7ebb33d20a-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.608128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.608038 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" Oct 30 22:00:24.608128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.608049 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw" event={"ID":"94f0c3cf-1c83-440a-a8e0-9288ae2b0672","Type":"ContainerDied","Data":"4a90e0db00a766f7cd4989c3b59b0a746fe75ad2017ff57dd1c0002e87796377"} Oct 30 22:00:24.608128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.608102 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a90e0db00a766f7cd4989c3b59b0a746fe75ad2017ff57dd1c0002e87796377" Oct 30 22:00:24.611751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.611719 5036 generic.go:356] "Generic (PLEG): container finished" podID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerID="30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f" exitCode=0 Oct 30 22:00:24.611897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.611761 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" Oct 30 22:00:24.611897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.611759 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerDied","Data":"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f"} Oct 30 22:00:24.611897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.611890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8" event={"ID":"f45a8d71-db14-47f4-a20e-7f7ebb33d20a","Type":"ContainerDied","Data":"2020c29ab615b4b276d2199f7ffb35a8b81d6abe10baa3a5690ef709a0eae9f2"} Oct 30 22:00:24.612055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.611922 5036 scope.go:122] "RemoveContainer" containerID="30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f" Oct 30 22:00:24.625090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.625053 5036 scope.go:122] "RemoveContainer" containerID="eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f" Oct 30 22:00:24.638898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.638817 5036 scope.go:122] "RemoveContainer" containerID="30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f" Oct 30 22:00:24.639323 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:24.639145 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f\": container with ID starting with 30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f not found: ID does not exist" containerID="30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f" Oct 30 22:00:24.639323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.639187 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f"} err="failed to get container status \"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f\": rpc error: code = NotFound desc = could not find container \"30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f\": container with ID starting with 30bde29c72358366efa57e9a97fd6984474c285dbffdac234add15f1d81f3a9f not found: ID does not exist" Oct 30 22:00:24.639323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.639213 5036 scope.go:122] "RemoveContainer" containerID="eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f" Oct 30 22:00:24.639545 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:24.639522 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f\": container with ID starting with eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f not found: ID does not exist" containerID="eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f" Oct 30 22:00:24.639664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.639548 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f"} err="failed to get container status \"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f\": rpc error: code = NotFound desc = could not find container \"eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f\": container with ID starting with eb950c42acda7f56f2baf0a8292919018d1590537465452e40d360319b3faf3f not found: ID does not exist" Oct 30 22:00:24.645667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.645640 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 22:00:24.652789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:24.652760 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8d2c42-openshift-default-6f6795dd6c-2b7z8"] Oct 30 22:00:25.635495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.635465 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:00:25.636688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636658 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.636688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636682 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636697 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636704 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636718 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636725 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636744 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636751 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636763 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636771 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636780 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636786 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636797 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636804 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636821 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636828 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636840 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636847 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636856 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636863 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636879 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:25.636881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636886 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636898 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636904 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636919 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636928 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636941 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636948 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636960 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636967 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636980 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.636987 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637004 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637010 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637021 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637028 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637039 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637046 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637068 5036 cpu_manager.go:414] "containerMap: removing container" podUID="94f0c3cf-1c83-440a-a8e0-9288ae2b0672" containerName="revoke-keys" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637076 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f0c3cf-1c83-440a-a8e0-9288ae2b0672" containerName="revoke-keys" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637089 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637096 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637123 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637130 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637352 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637368 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637381 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637393 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637404 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f0c3cf-1c83-440a-a8e0-9288ae2b0672" containerName="revoke-keys" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637416 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637425 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637434 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637443 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637452 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637460 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1036b5-a293-4661-b033-7eab36c87224" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637471 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637481 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637490 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637499 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637509 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637519 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637529 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637537 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637545 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637555 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:25.637920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.637564 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:25.644321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.644251 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.667331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.667256 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:00:25.736294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.723304 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ecffad-dockercfg-8vb22\"" Oct 30 22:00:25.736294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.723409 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ecffad\"" Oct 30 22:00:25.784289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.771620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnvg\" (UniqueName: \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.784289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.771733 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.876322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.875056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnvg\" (UniqueName: \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.876322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.875149 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.876823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.876797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:25.890241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.890175 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:00:25.890696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.890679 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:00:25.890988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:25.890967 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnvg\" (UniqueName: \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\") pod \"payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:26.009069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.009036 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:26.253018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.252982 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" path="/var/lib/kubelet/pods/f45a8d71-db14-47f4-a20e-7f7ebb33d20a/volumes" Oct 30 22:00:26.389683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.389650 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:00:26.391397 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:26.391373 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61ad52d_9c2f_48b9_aeaf_dc50d113b70b.slice/crio-cc338e38ce96bba3586a4abb2de9f4dc28a1b80e3e4876d7ce6cca233e43e5de WatchSource:0}: Error finding container cc338e38ce96bba3586a4abb2de9f4dc28a1b80e3e4876d7ce6cca233e43e5de: Status 404 returned error can't find the container with id cc338e38ce96bba3586a4abb2de9f4dc28a1b80e3e4876d7ce6cca233e43e5de Oct 30 22:00:26.652756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.652718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" event={"ID":"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b","Type":"ContainerStarted","Data":"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf"} Oct 30 22:00:26.653289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.652784 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" event={"ID":"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b","Type":"ContainerStarted","Data":"cc338e38ce96bba3586a4abb2de9f4dc28a1b80e3e4876d7ce6cca233e43e5de"} Oct 30 22:00:26.653289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.652951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:26.683427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:26.683382 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" podStartSLOduration=1.6833671369999998 podStartE2EDuration="1.683367137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:26.673994976 +0000 UTC m=+1250.623949922" watchObservedRunningTime="2026-10-30 22:00:26.683367137 +0000 UTC m=+1250.633322082" Oct 30 22:00:28.678739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.678704 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq"] Oct 30 22:00:28.679480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679464 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679481 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8e006c-38db-4d8f-abe6-dbc14851195b" containerName="cleanup" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679495 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679500 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45a8d71-db14-47f4-a20e-7f7ebb33d20a" containerName="istio-proxy" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679514 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679520 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:28.679542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679543 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679549 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679581 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679586 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679694 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f284e515-5326-487f-8c53-f7c87506bae2" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679709 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa478d93-fc7a-4a9b-a3dc-469524500757" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679715 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcc2179-6acc-4de1-9573-75a113690029" containerName="cleanup" Oct 30 22:00:28.679726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679724 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4207f3dc-e5c9-4a91-9e71-34b83533046c" containerName="cleanup" Oct 30 22:00:28.679940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.679731 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a815ffc-c527-4219-a8dc-518b21c2f4cc" containerName="cleanup" Oct 30 22:00:28.683251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.683230 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.687375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.687354 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:28.687473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.687435 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:28.687519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.687502 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:00:28.687717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.687696 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:00:28.692128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.692102 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq"] Oct 30 22:00:28.879775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.879748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqq7k\" (UniqueName: \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.879943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.879879 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.880025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.880009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.981239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.981153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqq7k\" (UniqueName: \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.981408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.981249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.981408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.981381 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.981928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.981897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.983901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.983878 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:28.993075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:28.993051 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqq7k\" (UniqueName: \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:29.005065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:29.005048 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:29.307363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:29.307338 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq"] Oct 30 22:00:29.307781 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:00:29.307754 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802d0aeb_9279_4e18_beb4_583abfb9c9a2.slice/crio-92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e WatchSource:0}: Error finding container 92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e: Status 404 returned error can't find the container with id 92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e Oct 30 22:00:29.709743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:29.709706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" event={"ID":"802d0aeb-9279-4e18-beb4-583abfb9c9a2","Type":"ContainerStarted","Data":"92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e"} Oct 30 22:00:29.741097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:29.741068 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 22:00:30.748290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:30.747874 5036 generic.go:356] "Generic (PLEG): container finished" podID="802d0aeb-9279-4e18-beb4-583abfb9c9a2" containerID="e9882f7bbee9a98c8d9833dcd30ffa41a831d8ac7249a3debc9073f43f4fcdb0" exitCode=0 Oct 30 22:00:30.785349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:30.761583 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" event={"ID":"802d0aeb-9279-4e18-beb4-583abfb9c9a2","Type":"ContainerDied","Data":"e9882f7bbee9a98c8d9833dcd30ffa41a831d8ac7249a3debc9073f43f4fcdb0"} Oct 30 22:00:32.075463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.075425 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:32.170937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.170899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\" (UniqueName: \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\") pod \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " Oct 30 22:00:32.171150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.171007 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\") pod \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " Oct 30 22:00:32.171150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.171112 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\") pod \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\" (UID: \"802d0aeb-9279-4e18-beb4-583abfb9c9a2\") " Oct 30 22:00:32.171875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.171832 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca" pod "802d0aeb-9279-4e18-beb4-583abfb9c9a2" (UID: "802d0aeb-9279-4e18-beb4-583abfb9c9a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:32.180458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.180430 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca" pod "802d0aeb-9279-4e18-beb4-583abfb9c9a2" (UID: "802d0aeb-9279-4e18-beb4-583abfb9c9a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:32.184114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.184086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k" pod "802d0aeb-9279-4e18-beb4-583abfb9c9a2" (UID: "802d0aeb-9279-4e18-beb4-583abfb9c9a2"). InnerVolumeSpecName "kube-api-access-dqq7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:32.274035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.274009 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqq7k\" (UniqueName: \"kubernetes.io/projected/802d0aeb-9279-4e18-beb4-583abfb9c9a2-kube-api-access-dqq7k\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:32.274499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.274482 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:32.274607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.274597 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802d0aeb-9279-4e18-beb4-583abfb9c9a2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:32.798589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.798559 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" Oct 30 22:00:32.798737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.798562 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq" event={"ID":"802d0aeb-9279-4e18-beb4-583abfb9c9a2","Type":"ContainerDied","Data":"92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e"} Oct 30 22:00:32.798737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:32.798687 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f629b60535c02fd02ee100f91ac78f974bf38be4b635716337fef48c4d1f7e" Oct 30 22:00:34.742428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:34.742380 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" podUID="db27bc44-3464-47a1-8426-e770ffe15eda" containerName="payload-processing" containerID="cri-o://38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416" gracePeriod=25 Oct 30 22:00:35.261247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.258645 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 22:00:35.407314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.407246 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\" (UniqueName: \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\") pod \"db27bc44-3464-47a1-8426-e770ffe15eda\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " Oct 30 22:00:35.407952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.407930 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\") pod \"db27bc44-3464-47a1-8426-e770ffe15eda\" (UID: \"db27bc44-3464-47a1-8426-e770ffe15eda\") " Oct 30 22:00:35.408560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.408445 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume" pod "db27bc44-3464-47a1-8426-e770ffe15eda" (UID: "db27bc44-3464-47a1-8426-e770ffe15eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:35.409336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.409315 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db27bc44-3464-47a1-8426-e770ffe15eda-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:35.446288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.438437 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2" pod "db27bc44-3464-47a1-8426-e770ffe15eda" (UID: "db27bc44-3464-47a1-8426-e770ffe15eda"). InnerVolumeSpecName "kube-api-access-dnpr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:35.545357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.545313 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpr2\" (UniqueName: \"kubernetes.io/projected/db27bc44-3464-47a1-8426-e770ffe15eda-kube-api-access-dnpr2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:35.863153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.863059 5036 generic.go:356] "Generic (PLEG): container finished" podID="db27bc44-3464-47a1-8426-e770ffe15eda" containerID="38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416" exitCode=0 Oct 30 22:00:35.863767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.863122 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" Oct 30 22:00:35.864010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.863144 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" event={"ID":"db27bc44-3464-47a1-8426-e770ffe15eda","Type":"ContainerDied","Data":"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416"} Oct 30 22:00:35.864094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.864064 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7" event={"ID":"db27bc44-3464-47a1-8426-e770ffe15eda","Type":"ContainerDied","Data":"fd4425f435f68e02b6842d9b0afae3cf242682507d39e4b021be4d472c2e58a1"} Oct 30 22:00:35.864155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.864087 5036 scope.go:122] "RemoveContainer" containerID="38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416" Oct 30 22:00:35.877824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.877806 5036 scope.go:122] "RemoveContainer" containerID="38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416" Oct 30 22:00:35.878134 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:35.878110 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416\": container with ID starting with 38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416 not found: ID does not exist" containerID="38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416" Oct 30 22:00:35.878216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.878139 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416"} err="failed to get container status \"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416\": rpc error: code = NotFound desc = could not find container \"38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416\": container with ID starting with 38bf5e00c7e95eb2360b7dc36b7dc9ba7efa32237553e87d9fb1ee497b6df416 not found: ID does not exist" Oct 30 22:00:35.891156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.891134 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 22:00:35.896467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:35.896446 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c6cf1-cddc4889f-7qdm7"] Oct 30 22:00:36.204371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:36.204325 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db27bc44-3464-47a1-8426-e770ffe15eda" path="/var/lib/kubelet/pods/db27bc44-3464-47a1-8426-e770ffe15eda/volumes" Oct 30 22:00:37.332457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.332408 5036 scope.go:122] "RemoveContainer" containerID="7ee2134d705528b20c2e31c89cefb900a7e61b3b1c78014aeee04fdd36305432" Oct 30 22:00:37.379019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.378722 5036 scope.go:122] "RemoveContainer" containerID="ca1dc89ed3321592743459e03dee08cce58b2024c188daddfac9704691da291d" Oct 30 22:00:37.393898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.393875 5036 scope.go:122] "RemoveContainer" containerID="3e8618f54b3bb858b0872734f4e2c96d1cba2e33832205c17f1328ddaca580fa" Oct 30 22:00:37.407484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.407464 5036 scope.go:122] "RemoveContainer" containerID="3013e0228b1198a736eee9f4172dc8c06d6870ccc21fbccf4cc3a357b153cd4e" Oct 30 22:00:37.421920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.421898 5036 scope.go:122] "RemoveContainer" containerID="1565b7287ab0b534ec6fa7238f951f751acffd6ea69539817181418ddf60f873" Oct 30 22:00:37.496449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.494224 5036 scope.go:122] "RemoveContainer" containerID="2d5bcfcc5515947d0d29cc94c59c5dae038cafddbefbd7c04b5874269d1c983b" Oct 30 22:00:37.531631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.531479 5036 scope.go:122] "RemoveContainer" containerID="5765e8d55dc4360d5b4c81b2bd996aaccfab0e5d13d7cc38277799fcb38829ba" Oct 30 22:00:37.553393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.553367 5036 scope.go:122] "RemoveContainer" containerID="e2efc80bbb7ffb3a186fa26038b7bb666289b868107fd0ff00d8270394ceb3ca" Oct 30 22:00:37.567331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.567258 5036 scope.go:122] "RemoveContainer" containerID="0d6ed918ca0bc60b633efd363e1d2e180e21e72b58e52073fd3c0a48cd1fa186" Oct 30 22:00:37.682182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:37.682161 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:00:42.009193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.009158 5036 generic.go:356] "Generic (PLEG): container finished" podID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerID="5689143d0cdb33a710fa4193c5616452bac9087765a96201bd880356e92e78af" exitCode=0 Oct 30 22:00:42.009798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.009236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerDied","Data":"5689143d0cdb33a710fa4193c5616452bac9087765a96201bd880356e92e78af"} Oct 30 22:00:42.009883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.009861 5036 scope.go:122] "RemoveContainer" containerID="5689143d0cdb33a710fa4193c5616452bac9087765a96201bd880356e92e78af" Oct 30 22:00:42.011822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.011787 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:42.011931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.011898 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:42.011972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:42.011948 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:43.046744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:43.046699 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerStarted","Data":"0d654cf0ef3207e7da3375b09e7331ef8a557e9e0e0a344da04e3c057ba4d866"} Oct 30 22:00:43.704199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:43.704162 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:43.705254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:43.705224 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:00:43.705411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:43.705314 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 22:00:44.705027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:44.704990 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:00:44.705522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:44.705054 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 22:00:45.705441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:45.705393 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:00:45.705919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:45.705470 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 22:00:46.151290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:46.141214 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:46.705296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:46.705240 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:00:46.705486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:46.705321 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 22:00:47.577894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:47.574026 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw"] Oct 30 22:00:47.592085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:47.581609 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c6cf1-zkgkw"] Oct 30 22:00:47.704401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:47.704369 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:00:47.704691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:47.704444 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 22:00:48.131092 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:48.130473 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:00:48.192085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.192058 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f0c3cf-1c83-440a-a8e0-9288ae2b0672" path="/var/lib/kubelet/pods/94f0c3cf-1c83-440a-a8e0-9288ae2b0672/volumes" Oct 30 22:00:48.710725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.710694 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:48.712318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.712285 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:00:48.848581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.848533 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 22:00:48.849067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.849033 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" podUID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" containerName="payload-processing" containerID="cri-o://a5b5348816f496d591ad2326ccd2a49954976d1339cada3f19fd8c26eae47fd1" gracePeriod=30 Oct 30 22:00:48.863510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.863474 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 22:00:48.863986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:48.863952 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" podUID="e894d868-4d71-4662-8559-02f26690c377" containerName="payload-pre-processing" containerID="cri-o://e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857" gracePeriod=30 Oct 30 22:00:49.187333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.187302 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq"] Oct 30 22:00:49.194666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.194644 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b3fe4-x75wq"] Oct 30 22:00:49.204372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.204340 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 22:00:49.212421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.212392 5036 generic.go:356] "Generic (PLEG): container finished" podID="e894d868-4d71-4662-8559-02f26690c377" containerID="e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857" exitCode=0 Oct 30 22:00:49.212539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.212502 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" Oct 30 22:00:49.212664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.212643 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" event={"ID":"e894d868-4d71-4662-8559-02f26690c377","Type":"ContainerDied","Data":"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857"} Oct 30 22:00:49.213700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.213666 5036 scope.go:122] "RemoveContainer" containerID="e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857" Oct 30 22:00:49.214013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.213368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc" event={"ID":"e894d868-4d71-4662-8559-02f26690c377","Type":"ContainerDied","Data":"9cc35947917bcb1afc476e8fcca04185cddb956f133fad404b7b7a8bf79ddc7d"} Oct 30 22:00:49.215051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.214933 5036 generic.go:356] "Generic (PLEG): container finished" podID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" containerID="a5b5348816f496d591ad2326ccd2a49954976d1339cada3f19fd8c26eae47fd1" exitCode=0 Oct 30 22:00:49.215454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.215428 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" event={"ID":"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f","Type":"ContainerDied","Data":"a5b5348816f496d591ad2326ccd2a49954976d1339cada3f19fd8c26eae47fd1"} Oct 30 22:00:49.292336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.290931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\") pod \"e894d868-4d71-4662-8559-02f26690c377\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " Oct 30 22:00:49.292336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.291180 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\" (UniqueName: \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\") pod \"e894d868-4d71-4662-8559-02f26690c377\" (UID: \"e894d868-4d71-4662-8559-02f26690c377\") " Oct 30 22:00:49.335245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.335099 5036 scope.go:122] "RemoveContainer" containerID="e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857" Oct 30 22:00:49.336681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.336641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume" pod "e894d868-4d71-4662-8559-02f26690c377" (UID: "e894d868-4d71-4662-8559-02f26690c377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:49.338778 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:00:49.338693 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857\": container with ID starting with e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857 not found: ID does not exist" containerID="e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857" Oct 30 22:00:49.338778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.338735 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857"} err="failed to get container status \"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857\": rpc error: code = NotFound desc = could not find container \"e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857\": container with ID starting with e648408dd5962eb97b8cfeb9bb693fc913419166028d8e7da1d080af75fa8857 not found: ID does not exist" Oct 30 22:00:49.373398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.371867 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq" pod "e894d868-4d71-4662-8559-02f26690c377" (UID: "e894d868-4d71-4662-8559-02f26690c377"). InnerVolumeSpecName "kube-api-access-782pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:49.455071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.455040 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 22:00:49.463893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.463856 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\" (UniqueName: \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\") pod \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " Oct 30 22:00:49.504296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.489599 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\") pod \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\" (UID: \"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f\") " Oct 30 22:00:49.504296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.491051 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume" pod "c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" (UID: "c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:49.504296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.494620 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-782pq\" (UniqueName: \"kubernetes.io/projected/e894d868-4d71-4662-8559-02f26690c377-kube-api-access-782pq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.504296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.494675 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e894d868-4d71-4662-8559-02f26690c377-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.504296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.498537 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852" pod "c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" (UID: "c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f"). InnerVolumeSpecName "kube-api-access-b8852". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:49.592430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.592394 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 22:00:49.596762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.596726 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8852\" (UniqueName: \"kubernetes.io/projected/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-kube-api-access-b8852\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.596762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.596756 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.614666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:49.614635 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-6c5657fbdc-phhlc"] Oct 30 22:00:50.201292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.201196 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802d0aeb-9279-4e18-beb4-583abfb9c9a2" path="/var/lib/kubelet/pods/802d0aeb-9279-4e18-beb4-583abfb9c9a2/volumes" Oct 30 22:00:50.201670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.201654 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e894d868-4d71-4662-8559-02f26690c377" path="/var/lib/kubelet/pods/e894d868-4d71-4662-8559-02f26690c377/volumes" Oct 30 22:00:50.224760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.224729 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" Oct 30 22:00:50.224760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.224753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj" event={"ID":"c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f","Type":"ContainerDied","Data":"0be00325b3e0349b29a991765e95aafb713b8bb0de612b2563fa4453d5ceb3ee"} Oct 30 22:00:50.224946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.224808 5036 scope.go:122] "RemoveContainer" containerID="a5b5348816f496d591ad2326ccd2a49954976d1339cada3f19fd8c26eae47fd1" Oct 30 22:00:50.253462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.253435 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 22:00:50.257845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.257821 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-77c79f4b58-569jj"] Oct 30 22:00:50.468762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.468670 5036 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podc5d9a1be-eddc-4de6-b600-1407aada133b"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podc5d9a1be-eddc-4de6-b600-1407aada133b] : Timed out while waiting for systemd to remove kubepods-burstable-podc5d9a1be_eddc_4de6_b600_1407aada133b.slice" Oct 30 22:00:50.468762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:50.468709 5036 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="c5d9a1be-eddc-4de6-b600-1407aada133b" err="failed to delete cgroup paths for [kubepods burstable podc5d9a1be-eddc-4de6-b600-1407aada133b] : unable to destroy cgroup paths for cgroup [kubepods burstable podc5d9a1be-eddc-4de6-b600-1407aada133b] : Timed out while waiting for systemd to remove kubepods-burstable-podc5d9a1be_eddc_4de6_b600_1407aada133b.slice" Oct 30 22:00:52.189200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:52.189164 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" path="/var/lib/kubelet/pods/c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f/volumes" Oct 30 22:00:59.733884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:59.733849 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 22:00:59.734462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:00:59.734104 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" containerID="cri-o://516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776" gracePeriod=30 Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.146713 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv"] Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147469 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147482 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147492 5036 cpu_manager.go:414] "containerMap: removing container" podUID="db27bc44-3464-47a1-8426-e770ffe15eda" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147497 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db27bc44-3464-47a1-8426-e770ffe15eda" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147506 5036 cpu_manager.go:414] "containerMap: removing container" podUID="802d0aeb-9279-4e18-beb4-583abfb9c9a2" containerName="revoke-keys" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147511 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802d0aeb-9279-4e18-beb4-583abfb9c9a2" containerName="revoke-keys" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147522 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e894d868-4d71-4662-8559-02f26690c377" containerName="payload-pre-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147527 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e894d868-4d71-4662-8559-02f26690c377" containerName="payload-pre-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147633 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b9c8e3-c4aa-4f1a-adea-677007e1ef8f" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147642 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="db27bc44-3464-47a1-8426-e770ffe15eda" containerName="payload-processing" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147648 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="802d0aeb-9279-4e18-beb4-583abfb9c9a2" containerName="revoke-keys" Oct 30 22:01:00.150315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.147654 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e894d868-4d71-4662-8559-02f26690c377" containerName="payload-pre-processing" Oct 30 22:01:00.154014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.153992 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.164133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.164102 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv"] Oct 30 22:01:00.165349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.165324 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:00.165976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.165955 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:00.167835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.167807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.168029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.167983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.168361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.168247 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvj4\" (UniqueName: \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.184908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.184856 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:01:00.185081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.185053 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:00.270096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.270066 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.270316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.270297 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.270397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.270381 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvj4\" (UniqueName: \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.270957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.270934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.272686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.272658 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.282780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.282756 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvj4\" (UniqueName: \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\") pod \"maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.498603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.498516 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:00.671781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.671750 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6"] Oct 30 22:01:00.677576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.677554 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.688123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.688096 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6"] Oct 30 22:01:00.782696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.782611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.782696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.782665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.783111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.782871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpst8\" (UniqueName: \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.791694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.791665 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv"] Oct 30 22:01:00.792224 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:00.792196 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9f2d2e_2d97_4799_8cdc_d8bb9215011d.slice/crio-dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3 WatchSource:0}: Error finding container dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3: Status 404 returned error can't find the container with id dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3 Oct 30 22:01:00.884772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.884739 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.884937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.884779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.884937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.884899 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpst8\" (UniqueName: \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.885508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.885484 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.887361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.887340 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.904911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.904879 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpst8\" (UniqueName: \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\") pod \"maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:00.995456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:00.995423 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:01.354342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:01.350014 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6"] Oct 30 22:01:01.392607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:01.392569 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" event={"ID":"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d","Type":"ContainerStarted","Data":"9295ca65dc686fb4c06c41c318948c98c4809d2565f33bad090572ff6a67ced8"} Oct 30 22:01:01.392770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:01.392638 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" event={"ID":"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d","Type":"ContainerStarted","Data":"dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3"} Oct 30 22:01:01.394341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:01.394296 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" event={"ID":"61960944-a833-4f97-9feb-0d5c5989da0c","Type":"ContainerStarted","Data":"84a054852de81dd225fc0771b8849c9cb738dc36000caeb3358e585e10a5c738"} Oct 30 22:01:01.415083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:01.415034 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" podStartSLOduration=1.415021574 podStartE2EDuration="1.415021574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:01.409606399 +0000 UTC m=+1285.359561350" watchObservedRunningTime="2026-10-30 22:01:01.415021574 +0000 UTC m=+1285.364976545" Oct 30 22:01:02.408900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:02.408857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" event={"ID":"61960944-a833-4f97-9feb-0d5c5989da0c","Type":"ContainerStarted","Data":"7dc1741b858bf37200ce4e2948a7bacb62e14f487b8d2edd3d7ae8d36749badf"} Oct 30 22:01:02.411821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:02.411278 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" containerID="9295ca65dc686fb4c06c41c318948c98c4809d2565f33bad090572ff6a67ced8" exitCode=0 Oct 30 22:01:02.411821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:02.411484 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" event={"ID":"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d","Type":"ContainerDied","Data":"9295ca65dc686fb4c06c41c318948c98c4809d2565f33bad090572ff6a67ced8"} Oct 30 22:01:02.435971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:02.435902 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" podStartSLOduration=2.435883396 podStartE2EDuration="2.435883396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:02.430796053 +0000 UTC m=+1286.380751001" watchObservedRunningTime="2026-10-30 22:01:02.435883396 +0000 UTC m=+1286.385838342" Oct 30 22:01:03.433000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.432962 5036 generic.go:356] "Generic (PLEG): container finished" podID="61960944-a833-4f97-9feb-0d5c5989da0c" containerID="7dc1741b858bf37200ce4e2948a7bacb62e14f487b8d2edd3d7ae8d36749badf" exitCode=0 Oct 30 22:01:03.433452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.433041 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" event={"ID":"61960944-a833-4f97-9feb-0d5c5989da0c","Type":"ContainerDied","Data":"7dc1741b858bf37200ce4e2948a7bacb62e14f487b8d2edd3d7ae8d36749badf"} Oct 30 22:01:03.630841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.630811 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:03.693632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.693539 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\" (UniqueName: \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\") pod \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " Oct 30 22:01:03.693632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.693607 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\") pod \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " Oct 30 22:01:03.693834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.693789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\") pod \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\" (UID: \"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d\") " Oct 30 22:01:03.694124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.694086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca" pod "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" (UID: "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:03.694608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.694584 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:03.696031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.696007 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca" pod "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" (UID: "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:03.696366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.696345 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4" pod "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" (UID: "6f9f2d2e-2d97-4799-8cdc-d8bb9215011d"). InnerVolumeSpecName "kube-api-access-9vvj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:03.795718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.795687 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvj4\" (UniqueName: \"kubernetes.io/projected/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-kube-api-access-9vvj4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:03.795718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:03.795717 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:04.445347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.445312 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" Oct 30 22:01:04.445831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.445311 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv" event={"ID":"6f9f2d2e-2d97-4799-8cdc-d8bb9215011d","Type":"ContainerDied","Data":"dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3"} Oct 30 22:01:04.445831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.445462 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcb3be9e48be48123d6a7b1badd37dd2b94abf56be9f91dea88c7276fba0bd3" Oct 30 22:01:04.640811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.640778 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:04.708666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.708635 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\") pod \"61960944-a833-4f97-9feb-0d5c5989da0c\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " Oct 30 22:01:04.708829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.708713 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\") pod \"61960944-a833-4f97-9feb-0d5c5989da0c\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " Oct 30 22:01:04.708829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.708753 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\" (UniqueName: \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\") pod \"61960944-a833-4f97-9feb-0d5c5989da0c\" (UID: \"61960944-a833-4f97-9feb-0d5c5989da0c\") " Oct 30 22:01:04.709135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.709110 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca" pod "61960944-a833-4f97-9feb-0d5c5989da0c" (UID: "61960944-a833-4f97-9feb-0d5c5989da0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:04.709526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.709507 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:04.711632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.711593 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8" pod "61960944-a833-4f97-9feb-0d5c5989da0c" (UID: "61960944-a833-4f97-9feb-0d5c5989da0c"). InnerVolumeSpecName "kube-api-access-fpst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:04.711724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.711643 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca" pod "61960944-a833-4f97-9feb-0d5c5989da0c" (UID: "61960944-a833-4f97-9feb-0d5c5989da0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:04.721438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.721412 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8_cc4940a5-b861-40d5-8c0e-6263d1cde106/payload-processing/0.log" Oct 30 22:01:04.723513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.723486 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:01:04.723626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.723538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:01:04.811126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.811097 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61960944-a833-4f97-9feb-0d5c5989da0c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:04.811126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.811123 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpst8\" (UniqueName: \"kubernetes.io/projected/61960944-a833-4f97-9feb-0d5c5989da0c-kube-api-access-fpst8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:04.954999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:04.948065 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9gpsn_fed55f70-e371-4b9a-9258-e70a781e919c/payload-processing/0.log" Oct 30 22:01:05.107012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.106986 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 22:01:05.193154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.193129 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9gpsn_fed55f70-e371-4b9a-9258-e70a781e919c/payload-processing/0.log" Oct 30 22:01:05.217053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217014 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217086 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217209 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217244 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217410 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.217660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.217569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\") pod \"6dbb1593-f067-4962-af30-67e3758a305d\" (UID: \"6dbb1593-f067-4962-af30-67e3758a305d\") " Oct 30 22:01:05.218527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218028 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.218527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218095 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.218527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218225 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.218527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218245 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.218527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218488 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.219023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.218898 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.219146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.219119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:05.221073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.221048 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "kube-api-access-k7cl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:05.221393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.221372 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:01:05.221864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.221841 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:05.222144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.222121 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy" pod "6dbb1593-f067-4962-af30-67e3758a305d" (UID: "6dbb1593-f067-4962-af30-67e3758a305d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.320374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320343 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cl8\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-kube-api-access-k7cl8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320376 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbb1593-f067-4962-af30-67e3758a305d-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320397 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbb1593-f067-4962-af30-67e3758a305d-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320424 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbb1593-f067-4962-af30-67e3758a305d-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320437 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320451 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.320641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.320467 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbb1593-f067-4962-af30-67e3758a305d-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.403891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.403862 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst_52b962ac-f06e-4e57-92a5-74248dcd4eb1/payload-processing/0.log" Oct 30 22:01:05.455900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.455876 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" Oct 30 22:01:05.456372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.455912 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6" event={"ID":"61960944-a833-4f97-9feb-0d5c5989da0c","Type":"ContainerDied","Data":"84a054852de81dd225fc0771b8849c9cb738dc36000caeb3358e585e10a5c738"} Oct 30 22:01:05.456372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.455971 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a054852de81dd225fc0771b8849c9cb738dc36000caeb3358e585e10a5c738" Oct 30 22:01:05.458075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.458044 5036 generic.go:356] "Generic (PLEG): container finished" podID="6dbb1593-f067-4962-af30-67e3758a305d" containerID="516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776" exitCode=0 Oct 30 22:01:05.458208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.458101 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" Oct 30 22:01:05.458208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.458127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerDied","Data":"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776"} Oct 30 22:01:05.458208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.458183 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6" event={"ID":"6dbb1593-f067-4962-af30-67e3758a305d","Type":"ContainerDied","Data":"dd8e86745354f653ba8cd46e250216710e11c3fb3a2a56433867412047d9b0c0"} Oct 30 22:01:05.458430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.458214 5036 scope.go:122] "RemoveContainer" containerID="516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776" Oct 30 22:01:05.471813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.471794 5036 scope.go:122] "RemoveContainer" containerID="c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f" Oct 30 22:01:05.482987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.482972 5036 scope.go:122] "RemoveContainer" containerID="516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776" Oct 30 22:01:05.483305 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:05.483282 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776\": container with ID starting with 516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776 not found: ID does not exist" containerID="516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776" Oct 30 22:01:05.483411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.483312 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776"} err="failed to get container status \"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776\": rpc error: code = NotFound desc = could not find container \"516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776\": container with ID starting with 516a6050b2626b0389813066ed2c3e4b6904f8365edbbfdb98e5ef978ff7d776 not found: ID does not exist" Oct 30 22:01:05.483411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.483330 5036 scope.go:122] "RemoveContainer" containerID="c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f" Oct 30 22:01:05.483625 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:05.483608 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f\": container with ID starting with c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f not found: ID does not exist" containerID="c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f" Oct 30 22:01:05.483668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.483632 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f"} err="failed to get container status \"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f\": rpc error: code = NotFound desc = could not find container \"c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f\": container with ID starting with c16ec69aaafc5567c0bf142eb9b9b54a2320efeda24052831417075fda54066f not found: ID does not exist" Oct 30 22:01:05.497530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.497508 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 22:01:05.503282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.503248 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0b3fe4-openshift-default-6469f6f47f-x7xm6"] Oct 30 22:01:05.594464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:05.594438 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst_52b962ac-f06e-4e57-92a5-74248dcd4eb1/payload-processing/0.log" Oct 30 22:01:06.189274 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.189240 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbb1593-f067-4962-af30-67e3758a305d" path="/var/lib/kubelet/pods/6dbb1593-f067-4962-af30-67e3758a305d/volumes" Oct 30 22:01:06.340897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.340863 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf"] Oct 30 22:01:06.341624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341607 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" containerName="revoke-keys" Oct 30 22:01:06.341624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341625 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" containerName="revoke-keys" Oct 30 22:01:06.341741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341652 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:06.341741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341657 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:06.341741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341664 5036 cpu_manager.go:414] "containerMap: removing container" podUID="61960944-a833-4f97-9feb-0d5c5989da0c" containerName="revoke-keys" Oct 30 22:01:06.341741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341670 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61960944-a833-4f97-9feb-0d5c5989da0c" containerName="revoke-keys" Oct 30 22:01:06.342060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341787 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" containerName="revoke-keys" Oct 30 22:01:06.342060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341800 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:06.342060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.341806 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="61960944-a833-4f97-9feb-0d5c5989da0c" containerName="revoke-keys" Oct 30 22:01:06.347065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.347047 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.352542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.352519 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:06.352645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.352620 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:06.358259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.358237 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf"] Oct 30 22:01:06.391200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.391175 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:06.391364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.391227 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:01:06.441421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.441339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.441421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.441396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgk5l\" (UniqueName: \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.441617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.441553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.542918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.542881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.543426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.542963 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgk5l\" (UniqueName: \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.543426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.543098 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.543804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.543786 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.546090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.546066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.556069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.556036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgk5l\" (UniqueName: \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\") pod \"maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:06.666307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:06.666277 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:07.112389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:07.112227 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf"] Oct 30 22:01:07.479706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:07.479675 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" event={"ID":"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c","Type":"ContainerStarted","Data":"d48c6d86fd12dab9c17b98ef2fd9134f64039b036330f0d84d6b7f0bda7b4a15"} Oct 30 22:01:08.490530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:08.490497 5036 generic.go:356] "Generic (PLEG): container finished" podID="3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" containerID="ef99dea0786344599ed578f826f87af9e7efd3b1db481bb7ff3cde03211add92" exitCode=0 Oct 30 22:01:08.490916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:08.490579 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" event={"ID":"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c","Type":"ContainerDied","Data":"ef99dea0786344599ed578f826f87af9e7efd3b1db481bb7ff3cde03211add92"} Oct 30 22:01:09.709002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.708978 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:09.814824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.814734 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\" (UniqueName: \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\") pod \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " Oct 30 22:01:09.815014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.814911 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\") pod \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " Oct 30 22:01:09.815014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.814999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\") pod \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\" (UID: \"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c\") " Oct 30 22:01:09.815479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.815452 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca" pod "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" (UID: "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:09.815616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.815602 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:09.817851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.817826 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca" pod "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" (UID: "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:09.817851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.817836 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l" pod "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" (UID: "3b3ce1aa-89f5-4dc3-9fab-184cd73e130c"). InnerVolumeSpecName "kube-api-access-kgk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:09.917326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.917297 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgk5l\" (UniqueName: \"kubernetes.io/projected/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-kube-api-access-kgk5l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:09.917326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:09.917322 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:10.522881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:10.522806 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" Oct 30 22:01:10.522881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:10.522836 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf" event={"ID":"3b3ce1aa-89f5-4dc3-9fab-184cd73e130c","Type":"ContainerDied","Data":"d48c6d86fd12dab9c17b98ef2fd9134f64039b036330f0d84d6b7f0bda7b4a15"} Oct 30 22:01:10.523056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:10.522893 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48c6d86fd12dab9c17b98ef2fd9134f64039b036330f0d84d6b7f0bda7b4a15" Oct 30 22:01:11.976852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:11.976819 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 22:01:11.977377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:11.977019 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" containerID="cri-o://0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f" gracePeriod=30 Oct 30 22:01:11.981861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:11.981836 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 22:01:11.982066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:11.982030 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" containerID="cri-o://b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7" gracePeriod=30 Oct 30 22:01:12.724622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:12.724584 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:01:12.724874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:12.724839 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" podUID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" containerName="payload-processing" containerID="cri-o://9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170" gracePeriod=30 Oct 30 22:01:12.734432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:12.734393 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:01:12.734708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:12.734677 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" podUID="b58b9493-f7d4-4cee-b938-b1192996e5a0" containerName="payload-pre-processing" containerID="cri-o://cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c" gracePeriod=30 Oct 30 22:01:13.208841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.205498 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv"] Oct 30 22:01:13.209365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.209221 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-19cab1-bsdmv"] Oct 30 22:01:13.267585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.267560 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:01:13.307705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.307621 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\" (UniqueName: \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\") pod \"b58b9493-f7d4-4cee-b938-b1192996e5a0\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " Oct 30 22:01:13.308215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.308154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\") pod \"b58b9493-f7d4-4cee-b938-b1192996e5a0\" (UID: \"b58b9493-f7d4-4cee-b938-b1192996e5a0\") " Oct 30 22:01:13.311988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.311637 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume" pod "b58b9493-f7d4-4cee-b938-b1192996e5a0" (UID: "b58b9493-f7d4-4cee-b938-b1192996e5a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:13.316947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.316901 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t" pod "b58b9493-f7d4-4cee-b938-b1192996e5a0" (UID: "b58b9493-f7d4-4cee-b938-b1192996e5a0"). InnerVolumeSpecName "kube-api-access-c9j7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:13.361549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.361519 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:01:13.411919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.411868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\" (UniqueName: \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\") pod \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " Oct 30 22:01:13.412110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.411947 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\") pod \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\" (UID: \"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4\") " Oct 30 22:01:13.412688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.412655 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume" pod "ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" (UID: "ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:13.412877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.412849 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j7t\" (UniqueName: \"kubernetes.io/projected/b58b9493-f7d4-4cee-b938-b1192996e5a0-kube-api-access-c9j7t\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.412961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.412888 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58b9493-f7d4-4cee-b938-b1192996e5a0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.412961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.412906 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.415627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.415599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4" pod "ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" (UID: "ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4"). InnerVolumeSpecName "kube-api-access-9p7w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:13.514459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.514360 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p7w4\" (UniqueName: \"kubernetes.io/projected/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4-kube-api-access-9p7w4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.552845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.552810 5036 generic.go:356] "Generic (PLEG): container finished" podID="b58b9493-f7d4-4cee-b938-b1192996e5a0" containerID="cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c" exitCode=0 Oct 30 22:01:13.553106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.552855 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" Oct 30 22:01:13.553106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.552886 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" event={"ID":"b58b9493-f7d4-4cee-b938-b1192996e5a0","Type":"ContainerDied","Data":"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c"} Oct 30 22:01:13.553106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.552961 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq" event={"ID":"b58b9493-f7d4-4cee-b938-b1192996e5a0","Type":"ContainerDied","Data":"009759dd462e494b436409843ddf189a30cd7679eda92d13777a25b3d7f93300"} Oct 30 22:01:13.553106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.552983 5036 scope.go:122] "RemoveContainer" containerID="cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c" Oct 30 22:01:13.557518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.557491 5036 generic.go:356] "Generic (PLEG): container finished" podID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" containerID="9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170" exitCode=0 Oct 30 22:01:13.557638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.557540 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" Oct 30 22:01:13.557638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.557541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" event={"ID":"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4","Type":"ContainerDied","Data":"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170"} Oct 30 22:01:13.557729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.557699 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h" event={"ID":"ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4","Type":"ContainerDied","Data":"bfe1e3352e47ad1989fb6dee2f392a5612b0dadd1777f58ebe3cc52fa79afdc8"} Oct 30 22:01:13.567192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.567166 5036 scope.go:122] "RemoveContainer" containerID="cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c" Oct 30 22:01:13.567733 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:13.567705 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c\": container with ID starting with cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c not found: ID does not exist" containerID="cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c" Oct 30 22:01:13.567884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.567746 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c"} err="failed to get container status \"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c\": rpc error: code = NotFound desc = could not find container \"cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c\": container with ID starting with cafc3aba9e81131dad6c71e1b13a59739f1ee557ad0369513a43a36b3267034c not found: ID does not exist" Oct 30 22:01:13.567884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.567766 5036 scope.go:122] "RemoveContainer" containerID="9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170" Oct 30 22:01:13.580103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.580076 5036 scope.go:122] "RemoveContainer" containerID="9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170" Oct 30 22:01:13.580501 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:13.580477 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170\": container with ID starting with 9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170 not found: ID does not exist" containerID="9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170" Oct 30 22:01:13.580594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.580511 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170"} err="failed to get container status \"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170\": rpc error: code = NotFound desc = could not find container \"9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170\": container with ID starting with 9497e6516ed0034891680edb5a69d502d8f810133723c20f1e2d32349ae84170 not found: ID does not exist" Oct 30 22:01:13.585726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.585683 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:01:13.589930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.589908 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-19cab1-5c7c66794c-cxhkq"] Oct 30 22:01:13.600421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.600393 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:01:13.605215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:13.605192 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-19cab1-696d897b66-vcw9h"] Oct 30 22:01:14.190258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:14.190225 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9f2d2e-2d97-4799-8cdc-d8bb9215011d" path="/var/lib/kubelet/pods/6f9f2d2e-2d97-4799-8cdc-d8bb9215011d/volumes" Oct 30 22:01:14.190763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:14.190747 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" path="/var/lib/kubelet/pods/ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4/volumes" Oct 30 22:01:14.191115 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:14.191102 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58b9493-f7d4-4cee-b938-b1192996e5a0" path="/var/lib/kubelet/pods/b58b9493-f7d4-4cee-b938-b1192996e5a0/volumes" Oct 30 22:01:17.345386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.344742 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 22:01:17.371771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.371738 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 22:01:17.402277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402231 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402289 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402429 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402499 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402645 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.402878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402716 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.402878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402750 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:17.402878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402786 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\") pod \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\" (UID: \"20764b52-2d02-4d45-b5cb-722bb0e8e2a6\") " Oct 30 22:01:17.403156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402877 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.403156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.402906 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.403156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.403092 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.409034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.408982 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:01:17.410567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410538 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.410887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410862 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410895 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410911 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410925 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410938 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410952 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.410967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.410966 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.419006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.418981 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:17.433594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.433470 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g" pod "20764b52-2d02-4d45-b5cb-722bb0e8e2a6" (UID: "20764b52-2d02-4d45-b5cb-722bb0e8e2a6"). InnerVolumeSpecName "kube-api-access-9dz8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:17.511732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.511732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.511732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511720 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511883 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.511899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.512019 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.512107 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\") pod \"59e4a593-16fc-4734-a2c2-e3771683cdc5\" (UID: \"59e4a593-16fc-4734-a2c2-e3771683cdc5\") " Oct 30 22:01:17.512421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.512383 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.513302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.513232 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.513302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.513293 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dz8g\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-kube-api-access-9dz8g\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.513474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.513313 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20764b52-2d02-4d45-b5cb-722bb0e8e2a6-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.514471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.514429 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:17.515243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.515191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.515447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.515417 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.515818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.515796 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.525644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.525601 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:17.526042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.525975 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:01:17.526335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.526290 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:17.528782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.528722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq" pod "59e4a593-16fc-4734-a2c2-e3771683cdc5" (UID: "59e4a593-16fc-4734-a2c2-e3771683cdc5"). InnerVolumeSpecName "kube-api-access-ng8vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:17.614649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614617 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8vq\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-kube-api-access-ng8vq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614646 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e4a593-16fc-4734-a2c2-e3771683cdc5-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614660 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614672 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614681 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614693 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e4a593-16fc-4734-a2c2-e3771683cdc5-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614705 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614715 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e4a593-16fc-4734-a2c2-e3771683cdc5-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:17.614863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614803 5036 generic.go:356] "Generic (PLEG): container finished" podID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerID="0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f" exitCode=0 Oct 30 22:01:17.615210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614900 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" Oct 30 22:01:17.615210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.614942 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" event={"ID":"59e4a593-16fc-4734-a2c2-e3771683cdc5","Type":"ContainerDied","Data":"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f"} Oct 30 22:01:17.615210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.615006 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl" event={"ID":"59e4a593-16fc-4734-a2c2-e3771683cdc5","Type":"ContainerDied","Data":"22cc799cee6a76eda530d9857c0d394bf1051101a57a1d6d1b3099a707cc0d23"} Oct 30 22:01:17.615210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.615037 5036 scope.go:122] "RemoveContainer" containerID="0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f" Oct 30 22:01:17.616927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.616890 5036 generic.go:356] "Generic (PLEG): container finished" podID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerID="b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7" exitCode=0 Oct 30 22:01:17.617060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.616953 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" Oct 30 22:01:17.617060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.617049 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" event={"ID":"20764b52-2d02-4d45-b5cb-722bb0e8e2a6","Type":"ContainerDied","Data":"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7"} Oct 30 22:01:17.617184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.617087 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj" event={"ID":"20764b52-2d02-4d45-b5cb-722bb0e8e2a6","Type":"ContainerDied","Data":"047f98304daf53e81eddaf767f5a310f383f8448bc122a4dab960db767d6638e"} Oct 30 22:01:17.630257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.630238 5036 scope.go:122] "RemoveContainer" containerID="0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f" Oct 30 22:01:17.630541 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:17.630520 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f\": container with ID starting with 0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f not found: ID does not exist" containerID="0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f" Oct 30 22:01:17.630598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.630550 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f"} err="failed to get container status \"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f\": rpc error: code = NotFound desc = could not find container \"0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f\": container with ID starting with 0911ed7646b84211729f2e978a84e4a452b86e86cf5a90e4f763b90f8669f23f not found: ID does not exist" Oct 30 22:01:17.630598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.630565 5036 scope.go:122] "RemoveContainer" containerID="b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7" Oct 30 22:01:17.645108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.645088 5036 scope.go:122] "RemoveContainer" containerID="b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7" Oct 30 22:01:17.645493 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:17.645450 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7\": container with ID starting with b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7 not found: ID does not exist" containerID="b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7" Oct 30 22:01:17.645614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.645501 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7"} err="failed to get container status \"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7\": rpc error: code = NotFound desc = could not find container \"b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7\": container with ID starting with b710e170c148cb7744008b9a938713f2e31463af407caa0ea11e6e676b1d4ac7 not found: ID does not exist" Oct 30 22:01:17.655337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.655317 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 22:01:17.662140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.662113 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-l58cl"] Oct 30 22:01:17.669465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.669443 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 22:01:17.680812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:17.680783 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-5hrbj"] Oct 30 22:01:18.111541 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:18.111504 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:01:18.191362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:18.191330 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" path="/var/lib/kubelet/pods/20764b52-2d02-4d45-b5cb-722bb0e8e2a6/volumes" Oct 30 22:01:18.192064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:18.192049 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" path="/var/lib/kubelet/pods/59e4a593-16fc-4734-a2c2-e3771683cdc5/volumes" Oct 30 22:01:19.343995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.343959 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 22:01:19.344640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.344356 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" podUID="714f0b86-70df-43b9-806b-201e32fc8d36" containerName="payload-processing" containerID="cri-o://c45a41b3a2150b8924f701c3dd71255c0e0ca3dd3f8f611fc34f6ddebe8d0f16" gracePeriod=30 Oct 30 22:01:19.372342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.361883 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 22:01:19.372342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.362136 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" podUID="3b6bac20-909d-4354-9f53-6cd63e65a781" containerName="payload-pre-processing" containerID="cri-o://d62ac5588cb0b6e90ddbe396498711e00e8246f1c86c992eb10c3681778a1896" gracePeriod=30 Oct 30 22:01:19.737660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.729512 5036 generic.go:356] "Generic (PLEG): container finished" podID="714f0b86-70df-43b9-806b-201e32fc8d36" containerID="c45a41b3a2150b8924f701c3dd71255c0e0ca3dd3f8f611fc34f6ddebe8d0f16" exitCode=0 Oct 30 22:01:19.737660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.729647 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" event={"ID":"714f0b86-70df-43b9-806b-201e32fc8d36","Type":"ContainerDied","Data":"c45a41b3a2150b8924f701c3dd71255c0e0ca3dd3f8f611fc34f6ddebe8d0f16"} Oct 30 22:01:19.792239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.792181 5036 generic.go:356] "Generic (PLEG): container finished" podID="3b6bac20-909d-4354-9f53-6cd63e65a781" containerID="d62ac5588cb0b6e90ddbe396498711e00e8246f1c86c992eb10c3681778a1896" exitCode=0 Oct 30 22:01:19.792461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.792277 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" event={"ID":"3b6bac20-909d-4354-9f53-6cd63e65a781","Type":"ContainerDied","Data":"d62ac5588cb0b6e90ddbe396498711e00e8246f1c86c992eb10c3681778a1896"} Oct 30 22:01:19.924573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.924538 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 22:01:19.934031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.933776 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6"] Oct 30 22:01:19.942480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.942437 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c614f9-gfmf6"] Oct 30 22:01:19.950466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:19.950438 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 22:01:20.106043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.105965 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\") pod \"3b6bac20-909d-4354-9f53-6cd63e65a781\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " Oct 30 22:01:20.106206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\") pod \"714f0b86-70df-43b9-806b-201e32fc8d36\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " Oct 30 22:01:20.106206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106179 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\" (UniqueName: \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\") pod \"3b6bac20-909d-4354-9f53-6cd63e65a781\" (UID: \"3b6bac20-909d-4354-9f53-6cd63e65a781\") " Oct 30 22:01:20.106410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106249 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\" (UniqueName: \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\") pod \"714f0b86-70df-43b9-806b-201e32fc8d36\" (UID: \"714f0b86-70df-43b9-806b-201e32fc8d36\") " Oct 30 22:01:20.106474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106405 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume" pod "3b6bac20-909d-4354-9f53-6cd63e65a781" (UID: "3b6bac20-909d-4354-9f53-6cd63e65a781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:20.106672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106639 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume" pod "714f0b86-70df-43b9-806b-201e32fc8d36" (UID: "714f0b86-70df-43b9-806b-201e32fc8d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:20.106849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106830 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f0b86-70df-43b9-806b-201e32fc8d36-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:20.106919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.106855 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6bac20-909d-4354-9f53-6cd63e65a781-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:20.108719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.108698 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk" pod "714f0b86-70df-43b9-806b-201e32fc8d36" (UID: "714f0b86-70df-43b9-806b-201e32fc8d36"). InnerVolumeSpecName "kube-api-access-bkmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:20.109467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.109442 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h" pod "3b6bac20-909d-4354-9f53-6cd63e65a781" (UID: "3b6bac20-909d-4354-9f53-6cd63e65a781"). InnerVolumeSpecName "kube-api-access-w9w5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:20.190071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.190039 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61960944-a833-4f97-9feb-0d5c5989da0c" path="/var/lib/kubelet/pods/61960944-a833-4f97-9feb-0d5c5989da0c/volumes" Oct 30 22:01:20.207936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.207912 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkmdk\" (UniqueName: \"kubernetes.io/projected/714f0b86-70df-43b9-806b-201e32fc8d36-kube-api-access-bkmdk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:20.208048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.207951 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9w5h\" (UniqueName: \"kubernetes.io/projected/3b6bac20-909d-4354-9f53-6cd63e65a781-kube-api-access-w9w5h\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:20.809642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.809611 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" Oct 30 22:01:20.809642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.809631 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw" event={"ID":"714f0b86-70df-43b9-806b-201e32fc8d36","Type":"ContainerDied","Data":"e0d1a1a96eb65ef98fa126b0fe8d226c21f1e76f5cfeea9bba046231fed7028b"} Oct 30 22:01:20.810168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.809707 5036 scope.go:122] "RemoveContainer" containerID="c45a41b3a2150b8924f701c3dd71255c0e0ca3dd3f8f611fc34f6ddebe8d0f16" Oct 30 22:01:20.811783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.811297 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" event={"ID":"3b6bac20-909d-4354-9f53-6cd63e65a781","Type":"ContainerDied","Data":"41f6ec5b6f58e3aaacd57e029b558bb18bb82d5e6309d990d296b61638bc59f9"} Oct 30 22:01:20.811783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.811307 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr" Oct 30 22:01:20.827360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.827339 5036 scope.go:122] "RemoveContainer" containerID="d62ac5588cb0b6e90ddbe396498711e00e8246f1c86c992eb10c3681778a1896" Oct 30 22:01:20.837878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.837836 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 22:01:20.844849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.844823 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c614f9-68657b9dfb-7n6mw"] Oct 30 22:01:20.860412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.860381 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 22:01:20.866796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:20.866771 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c614f9-5ff76589bc-4lwtr"] Oct 30 22:01:21.448152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448118 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg"] Oct 30 22:01:21.448897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448876 5036 cpu_manager.go:414] "containerMap: removing container" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448900 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448926 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448933 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448946 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" containerName="payload-processing" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448952 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" containerName="payload-processing" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448964 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b58b9493-f7d4-4cee-b938-b1192996e5a0" containerName="payload-pre-processing" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448970 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58b9493-f7d4-4cee-b938-b1192996e5a0" containerName="payload-pre-processing" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448991 5036 cpu_manager.go:414] "containerMap: removing container" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.448997 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449004 5036 cpu_manager.go:414] "containerMap: removing container" podUID="714f0b86-70df-43b9-806b-201e32fc8d36" containerName="payload-processing" Oct 30 22:01:21.449005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449008 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714f0b86-70df-43b9-806b-201e32fc8d36" containerName="payload-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449022 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3b6bac20-909d-4354-9f53-6cd63e65a781" containerName="payload-pre-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449027 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6bac20-909d-4354-9f53-6cd63e65a781" containerName="payload-pre-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449034 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" containerName="revoke-keys" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449040 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" containerName="revoke-keys" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449176 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="20764b52-2d02-4d45-b5cb-722bb0e8e2a6" containerName="istio-proxy" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449188 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbb1593-f067-4962-af30-67e3758a305d" containerName="istio-proxy" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449195 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6e26f8-301b-4f3a-8c4d-a43a2db9c0c4" containerName="payload-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449201 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58b9493-f7d4-4cee-b938-b1192996e5a0" containerName="payload-pre-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449211 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e4a593-16fc-4734-a2c2-e3771683cdc5" containerName="istio-proxy" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449217 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="714f0b86-70df-43b9-806b-201e32fc8d36" containerName="payload-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449225 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6bac20-909d-4354-9f53-6cd63e65a781" containerName="payload-pre-processing" Oct 30 22:01:21.449477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.449231 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" containerName="revoke-keys" Oct 30 22:01:21.454326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.454306 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.461062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.461000 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:21.461456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.461436 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:21.461753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.461734 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:21.462007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.461986 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:01:21.467045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.467017 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg"] Oct 30 22:01:21.662698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.648300 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsvj\" (UniqueName: \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.663151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.663117 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.663543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.663521 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.765581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.765496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.765735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.765592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsvj\" (UniqueName: \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.765735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.765677 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.766210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.766191 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.768358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.768335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:21.777618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:21.777594 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsvj\" (UniqueName: \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\") pod \"maas-api-revoke-sub-bc186aaf-25gkg\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:22.070190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.070114 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:22.191695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.191652 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6bac20-909d-4354-9f53-6cd63e65a781" path="/var/lib/kubelet/pods/3b6bac20-909d-4354-9f53-6cd63e65a781/volumes" Oct 30 22:01:22.192203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.192180 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714f0b86-70df-43b9-806b-201e32fc8d36" path="/var/lib/kubelet/pods/714f0b86-70df-43b9-806b-201e32fc8d36/volumes" Oct 30 22:01:22.379990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.379962 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg"] Oct 30 22:01:22.380422 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:22.380389 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ee8749_ccec_421e_a955_871487de1de1.slice/crio-139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a WatchSource:0}: Error finding container 139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a: Status 404 returned error can't find the container with id 139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a Oct 30 22:01:22.836778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.836687 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" event={"ID":"b0ee8749-ccec-421e-a955-871487de1de1","Type":"ContainerStarted","Data":"63882080d85ca2256b9f170680d57ba945a4a8dc9cf8a7ca8dc99644ccdd36a3"} Oct 30 22:01:22.836778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.836760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" event={"ID":"b0ee8749-ccec-421e-a955-871487de1de1","Type":"ContainerStarted","Data":"139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a"} Oct 30 22:01:22.861041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:22.860993 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" podStartSLOduration=1.860978805 podStartE2EDuration="1.860978805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:22.858544067 +0000 UTC m=+1306.808499004" watchObservedRunningTime="2026-10-30 22:01:22.860978805 +0000 UTC m=+1306.810933751" Oct 30 22:01:23.005587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.005501 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:01:23.010351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.010323 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.017474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.017451 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c097ae-openshift-default-dockercfg-m7b4n\"" Oct 30 22:01:23.037339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.037311 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:01:23.083796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.083765 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.083838 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.083888 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpnr\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.083975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.083999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.084024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.084141 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.084293 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.084455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.084370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.188623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.188576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpnr\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189339 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189364 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189449 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189740 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.189947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.189931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.190070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.190055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.190241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.190209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.191077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.191053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.191639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.191617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.191867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.191848 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.192062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.192043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.222303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.207040 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.233923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.233854 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.245288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.242020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpnr\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.245288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.242241 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\") pod \"e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.342921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.340700 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:23.671148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.671119 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:01:23.672070 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:23.672045 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c5155a_5f2e_424e_bf79_4cd926c87347.slice/crio-46e5d04911ffd21ff41ff08d024313941ecd5652b2cd91b668c113d117b8fdae WatchSource:0}: Error finding container 46e5d04911ffd21ff41ff08d024313941ecd5652b2cd91b668c113d117b8fdae: Status 404 returned error can't find the container with id 46e5d04911ffd21ff41ff08d024313941ecd5652b2cd91b668c113d117b8fdae Oct 30 22:01:23.675149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.675111 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:23.675237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.675184 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:23.675237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.675222 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:23.850883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.850844 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerStarted","Data":"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082"} Oct 30 22:01:23.851093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.850912 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerStarted","Data":"46e5d04911ffd21ff41ff08d024313941ecd5652b2cd91b668c113d117b8fdae"} Oct 30 22:01:23.853254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.853158 5036 generic.go:356] "Generic (PLEG): container finished" podID="b0ee8749-ccec-421e-a955-871487de1de1" containerID="63882080d85ca2256b9f170680d57ba945a4a8dc9cf8a7ca8dc99644ccdd36a3" exitCode=0 Oct 30 22:01:23.853254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.853189 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" event={"ID":"b0ee8749-ccec-421e-a955-871487de1de1","Type":"ContainerDied","Data":"63882080d85ca2256b9f170680d57ba945a4a8dc9cf8a7ca8dc99644ccdd36a3"} Oct 30 22:01:23.879938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:23.879887 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podStartSLOduration=1.879874546 podStartE2EDuration="1.879874546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:23.874371841 +0000 UTC m=+1307.824326818" watchObservedRunningTime="2026-10-30 22:01:23.879874546 +0000 UTC m=+1307.829829491" Oct 30 22:01:24.164012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.163979 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm"] Oct 30 22:01:24.168763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.168742 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.190911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.190886 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm"] Oct 30 22:01:24.211212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.211180 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvg8\" (UniqueName: \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.211475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.211447 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.211842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.211821 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.312832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.312797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.313001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.312984 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.313118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.313103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvg8\" (UniqueName: \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.313725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.313698 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.315693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.315668 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.329207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.329177 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvg8\" (UniqueName: \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\") pod \"maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.341077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.341056 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:24.346981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.346960 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:24.487181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.487101 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:24.863391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.863258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:24.864737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.864709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:01:24.875229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:24.875211 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm"] Oct 30 22:01:24.877082 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:24.877050 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488ab854_f35c_4872_9f1a_18290dfb9979.slice/crio-f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f WatchSource:0}: Error finding container f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f: Status 404 returned error can't find the container with id f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f Oct 30 22:01:25.083802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.083776 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:25.136064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.136038 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\" (UniqueName: \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\") pod \"b0ee8749-ccec-421e-a955-871487de1de1\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " Oct 30 22:01:25.136245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.136225 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\") pod \"b0ee8749-ccec-421e-a955-871487de1de1\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " Oct 30 22:01:25.136420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.136353 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\") pod \"b0ee8749-ccec-421e-a955-871487de1de1\" (UID: \"b0ee8749-ccec-421e-a955-871487de1de1\") " Oct 30 22:01:25.136901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.136868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca" pod "b0ee8749-ccec-421e-a955-871487de1de1" (UID: "b0ee8749-ccec-421e-a955-871487de1de1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:25.137046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.137029 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:25.155525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.155497 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj" pod "b0ee8749-ccec-421e-a955-871487de1de1" (UID: "b0ee8749-ccec-421e-a955-871487de1de1"). InnerVolumeSpecName "kube-api-access-dtsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:25.156793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.156768 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca" pod "b0ee8749-ccec-421e-a955-871487de1de1" (UID: "b0ee8749-ccec-421e-a955-871487de1de1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:25.239070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.239042 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ee8749-ccec-421e-a955-871487de1de1-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:25.239070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.239068 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtsvj\" (UniqueName: \"kubernetes.io/projected/b0ee8749-ccec-421e-a955-871487de1de1-kube-api-access-dtsvj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:25.885768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.885716 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" event={"ID":"488ab854-f35c-4872-9f1a-18290dfb9979","Type":"ContainerStarted","Data":"546f18ce657a6f1d90af02f496b40f5a8c7f0729231ca2a24925ce3b9bcbeed9"} Oct 30 22:01:25.885981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.885836 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" event={"ID":"488ab854-f35c-4872-9f1a-18290dfb9979","Type":"ContainerStarted","Data":"f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f"} Oct 30 22:01:25.891315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.888784 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" Oct 30 22:01:25.891315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.888870 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg" event={"ID":"b0ee8749-ccec-421e-a955-871487de1de1","Type":"ContainerDied","Data":"139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a"} Oct 30 22:01:25.891315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.888920 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="139b9a19aa0134b1a48713e735bf3d6bc073a2644dfdb427cf8a8f56c99ea25a" Oct 30 22:01:25.915947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:25.915901 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" podStartSLOduration=1.915887136 podStartE2EDuration="1.915887136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:25.911822768 +0000 UTC m=+1309.861777715" watchObservedRunningTime="2026-10-30 22:01:25.915887136 +0000 UTC m=+1309.865842133" Oct 30 22:01:26.900087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:26.900048 5036 generic.go:356] "Generic (PLEG): container finished" podID="488ab854-f35c-4872-9f1a-18290dfb9979" containerID="546f18ce657a6f1d90af02f496b40f5a8c7f0729231ca2a24925ce3b9bcbeed9" exitCode=0 Oct 30 22:01:26.900521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:26.900124 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" event={"ID":"488ab854-f35c-4872-9f1a-18290dfb9979","Type":"ContainerDied","Data":"546f18ce657a6f1d90af02f496b40f5a8c7f0729231ca2a24925ce3b9bcbeed9"} Oct 30 22:01:27.941607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.941569 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:01:27.942307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.942257 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b0ee8749-ccec-421e-a955-871487de1de1" containerName="revoke-keys" Oct 30 22:01:27.942307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.942290 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ee8749-ccec-421e-a955-871487de1de1" containerName="revoke-keys" Oct 30 22:01:27.942461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.942446 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ee8749-ccec-421e-a955-871487de1de1" containerName="revoke-keys" Oct 30 22:01:27.946850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.946826 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:27.952234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.952197 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c097ae-dockercfg-vbkx4\"" Oct 30 22:01:27.962979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.962953 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:01:27.970747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:27.970710 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c097ae\"" Oct 30 22:01:28.041296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.040336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.041296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.040577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xxl\" (UniqueName: \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.190447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.187638 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xxl\" (UniqueName: \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.190447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.187782 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.190447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.188761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.250670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.239907 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:01:28.260628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.257228 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.260628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.260101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xxl\" (UniqueName: \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\") pod \"payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.270276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.270226 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.295345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.290615 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mp9\" (UniqueName: \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.295345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.290845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.300629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.300601 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:01:28.393119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.393084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mp9\" (UniqueName: \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.393249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.393232 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.394194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.394169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.404784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.404753 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:28.409726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.409702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mp9\" (UniqueName: \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\") pod \"payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.494526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.494440 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\") pod \"488ab854-f35c-4872-9f1a-18290dfb9979\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " Oct 30 22:01:28.494701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.494654 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\") pod \"488ab854-f35c-4872-9f1a-18290dfb9979\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " Oct 30 22:01:28.494774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.494722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\" (UniqueName: \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\") pod \"488ab854-f35c-4872-9f1a-18290dfb9979\" (UID: \"488ab854-f35c-4872-9f1a-18290dfb9979\") " Oct 30 22:01:28.495006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.494978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca" pod "488ab854-f35c-4872-9f1a-18290dfb9979" (UID: "488ab854-f35c-4872-9f1a-18290dfb9979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:28.495206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.495187 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.496768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.496746 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca" pod "488ab854-f35c-4872-9f1a-18290dfb9979" (UID: "488ab854-f35c-4872-9f1a-18290dfb9979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:28.496956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.496936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8" pod "488ab854-f35c-4872-9f1a-18290dfb9979" (UID: "488ab854-f35c-4872-9f1a-18290dfb9979"). InnerVolumeSpecName "kube-api-access-mlvg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:28.596961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.596929 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvg8\" (UniqueName: \"kubernetes.io/projected/488ab854-f35c-4872-9f1a-18290dfb9979-kube-api-access-mlvg8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.596961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.596958 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488ab854-f35c-4872-9f1a-18290dfb9979-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.611355 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:28.611316 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded3a111_52af_4ca7_beea_2779769345f6.slice/crio-5b8f1eb70c53e7aed12f6d444854d13bad0f3902692d367ac9ac6ae66e473b5d WatchSource:0}: Error finding container 5b8f1eb70c53e7aed12f6d444854d13bad0f3902692d367ac9ac6ae66e473b5d: Status 404 returned error can't find the container with id 5b8f1eb70c53e7aed12f6d444854d13bad0f3902692d367ac9ac6ae66e473b5d Oct 30 22:01:28.611615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.611588 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:01:28.614935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.614916 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:28.922313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.922241 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" event={"ID":"ded3a111-52af-4ca7-beea-2779769345f6","Type":"ContainerStarted","Data":"3e1ab81c1b8cc15549ee4a63ac63696f95e151cbcb72db89badbfc25652f9bfd"} Oct 30 22:01:28.922514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.922326 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" event={"ID":"ded3a111-52af-4ca7-beea-2779769345f6","Type":"ContainerStarted","Data":"5b8f1eb70c53e7aed12f6d444854d13bad0f3902692d367ac9ac6ae66e473b5d"} Oct 30 22:01:28.922514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.922359 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:01:28.922514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.922415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:28.924131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.924100 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" Oct 30 22:01:28.924309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.924125 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm" event={"ID":"488ab854-f35c-4872-9f1a-18290dfb9979","Type":"ContainerDied","Data":"f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f"} Oct 30 22:01:28.924309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.924167 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ce199d902280c572a78f8c73977b2be20e21e5a1a5ef4cabea89943f5aa91f" Oct 30 22:01:28.959819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:28.959774 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" podStartSLOduration=1.959759308 podStartE2EDuration="1.959759308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:28.959638096 +0000 UTC m=+1312.909593043" watchObservedRunningTime="2026-10-30 22:01:28.959759308 +0000 UTC m=+1312.909714254" Oct 30 22:01:29.060989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:29.060953 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw"] Oct 30 22:01:29.067816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:29.067788 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2685ea84-n8qvw"] Oct 30 22:01:30.022705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.022666 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" event={"ID":"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb","Type":"ContainerStarted","Data":"7518dad7e7192d780fd4777e3b4112a2c31bae9e82c61d09b9ef7e1b8eb01ec2"} Oct 30 22:01:30.023304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.023254 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" event={"ID":"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb","Type":"ContainerStarted","Data":"8f213726de783a9c763c677f2a86de6dddfba76d6e1fb8df3f3b9c2b54977d9a"} Oct 30 22:01:30.023489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.023475 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:30.083473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.082831 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" podStartSLOduration=3.082805272 podStartE2EDuration="3.082805272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:30.078156803 +0000 UTC m=+1314.028111748" watchObservedRunningTime="2026-10-30 22:01:30.082805272 +0000 UTC m=+1314.032760251" Oct 30 22:01:30.190798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.190767 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22514316-a5c7-439e-8a87-b36ca39af526" path="/var/lib/kubelet/pods/22514316-a5c7-439e-8a87-b36ca39af526/volumes" Oct 30 22:01:30.398842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.398813 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8"] Oct 30 22:01:30.399590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.399572 5036 cpu_manager.go:414] "containerMap: removing container" podUID="488ab854-f35c-4872-9f1a-18290dfb9979" containerName="revoke-keys" Oct 30 22:01:30.399642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.399590 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488ab854-f35c-4872-9f1a-18290dfb9979" containerName="revoke-keys" Oct 30 22:01:30.399714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.399705 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="488ab854-f35c-4872-9f1a-18290dfb9979" containerName="revoke-keys" Oct 30 22:01:30.403060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.403042 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.416566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.416544 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:30.416700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.416601 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:30.416979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.416966 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:01:30.417034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.416991 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:30.429394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.429369 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8"] Oct 30 22:01:30.454529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.454494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.454739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.454714 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.454853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.454834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.556667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.556629 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.556833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.556815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.556919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.556903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.557422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.557381 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.559480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.559457 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.569178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.569158 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:30.719119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:30.719037 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:31.013203 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:31.013169 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b714b5_577a_44c3_9ebb_3ec59b480bed.slice/crio-d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d WatchSource:0}: Error finding container d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d: Status 404 returned error can't find the container with id d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d Oct 30 22:01:31.013433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:31.013404 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8"] Oct 30 22:01:32.039866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.039831 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj"] Oct 30 22:01:32.044961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.044903 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be591507-zlhmj"] Oct 30 22:01:32.046490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.046457 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" event={"ID":"46b714b5-577a-44c3-9ebb-3ec59b480bed","Type":"ContainerStarted","Data":"8bde832068a1bbfe21c10f39f71ba804b945033014cdf036b182905e0f6eb12e"} Oct 30 22:01:32.046629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.046555 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" event={"ID":"46b714b5-577a-44c3-9ebb-3ec59b480bed","Type":"ContainerStarted","Data":"d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d"} Oct 30 22:01:32.189452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.189421 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d55bb0-c5ba-45f1-9d18-a123f22a22c8" path="/var/lib/kubelet/pods/42d55bb0-c5ba-45f1-9d18-a123f22a22c8/volumes" Oct 30 22:01:32.314341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.307078 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 22:01:32.419962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.419927 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:01:32.425896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.425864 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.436950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.436916 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a1ead7-openshift-default-dockercfg-7jn77\"" Oct 30 22:01:32.446691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.446656 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:01:32.493706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493673 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.493846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493714 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.493846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493816 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.493846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493843 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.493975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493956 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.494013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.493992 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.494108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.494092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.494141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.494125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.494220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.494197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4fs\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.607373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.597695 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.597250 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.611799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.611881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4fs\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612028 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612064 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612169 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612217 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.612391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.613244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.614049 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.618443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.616105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.643290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.636024 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.656081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.656050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4fs\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.707302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.703813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.713360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.712146 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.735373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.732669 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\") pod \"e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.758253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.757990 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:32.876304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.875594 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf"] Oct 30 22:01:32.880716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:32.880626 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3654d5-2sbsf"] Oct 30 22:01:33.066080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.066049 5036 generic.go:356] "Generic (PLEG): container finished" podID="46b714b5-577a-44c3-9ebb-3ec59b480bed" containerID="8bde832068a1bbfe21c10f39f71ba804b945033014cdf036b182905e0f6eb12e" exitCode=0 Oct 30 22:01:33.066538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.066109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" event={"ID":"46b714b5-577a-44c3-9ebb-3ec59b480bed","Type":"ContainerDied","Data":"8bde832068a1bbfe21c10f39f71ba804b945033014cdf036b182905e0f6eb12e"} Oct 30 22:01:33.066538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.066312 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" podUID="cc4940a5-b861-40d5-8c0e-6263d1cde106" containerName="payload-processing" containerID="cri-o://a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582" gracePeriod=30 Oct 30 22:01:33.121452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.116902 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:01:33.121452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.118534 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:33.121452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.118605 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:33.121452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.118634 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:33.257781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.257752 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:33.348599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.348568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\") pod \"46b714b5-577a-44c3-9ebb-3ec59b480bed\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " Oct 30 22:01:33.348822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.348805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\") pod \"46b714b5-577a-44c3-9ebb-3ec59b480bed\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " Oct 30 22:01:33.348884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.348869 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\") pod \"46b714b5-577a-44c3-9ebb-3ec59b480bed\" (UID: \"46b714b5-577a-44c3-9ebb-3ec59b480bed\") " Oct 30 22:01:33.349912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.349881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca" pod "46b714b5-577a-44c3-9ebb-3ec59b480bed" (UID: "46b714b5-577a-44c3-9ebb-3ec59b480bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:33.351693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.351666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca" pod "46b714b5-577a-44c3-9ebb-3ec59b480bed" (UID: "46b714b5-577a-44c3-9ebb-3ec59b480bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:33.352698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.352678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5" pod "46b714b5-577a-44c3-9ebb-3ec59b480bed" (UID: "46b714b5-577a-44c3-9ebb-3ec59b480bed"). InnerVolumeSpecName "kube-api-access-k9ps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:33.450992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.450951 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:33.450992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.450986 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/46b714b5-577a-44c3-9ebb-3ec59b480bed-kube-api-access-k9ps5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:33.451223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:33.451002 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b714b5-577a-44c3-9ebb-3ec59b480bed-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:34.076333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.076295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerStarted","Data":"f70b851585945a87dbe6f4e661b13826a02c64c7ced1f255dfac15ee6acc925a"} Oct 30 22:01:34.076850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.076364 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerStarted","Data":"a9a0db5d15e1957cbb8fa241d9f5a8d7671c14a9146a195db390672a94348d40"} Oct 30 22:01:34.078176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.078145 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" Oct 30 22:01:34.078326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.078196 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8" event={"ID":"46b714b5-577a-44c3-9ebb-3ec59b480bed","Type":"ContainerDied","Data":"d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d"} Oct 30 22:01:34.078326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.078235 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d745496921a2467ec857eab595ee42bcc60310eee65e94b39a5a71de12c4394d" Oct 30 22:01:34.138033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.137951 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podStartSLOduration=2.137934755 podStartE2EDuration="2.137934755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:34.136321771 +0000 UTC m=+1318.086276721" watchObservedRunningTime="2026-10-30 22:01:34.137934755 +0000 UTC m=+1318.087889702" Oct 30 22:01:34.189230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.189198 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3ce1aa-89f5-4dc3-9fab-184cd73e130c" path="/var/lib/kubelet/pods/3b3ce1aa-89f5-4dc3-9fab-184cd73e130c/volumes" Oct 30 22:01:34.402771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.402741 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:01:34.762350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.761593 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:34.852291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:34.851707 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:35.088707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:35.088630 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:35.089636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:35.089617 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:01:37.790509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.790480 5036 scope.go:122] "RemoveContainer" containerID="4aeb3f4cd10649b94741a12a662be99469e820c9bbdf8d27ad1514ab0cc23082" Oct 30 22:01:37.803252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.803219 5036 scope.go:122] "RemoveContainer" containerID="638c7393a8ab11379efb2dc07a9cbdd4eb124d8385214f117291429e0a8f384e" Oct 30 22:01:37.814216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.814195 5036 scope.go:122] "RemoveContainer" containerID="70d14df59e8803074bbb784ef2dee9f8f95fb8ef35391490396cf025d2380ce0" Oct 30 22:01:37.825876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.825854 5036 scope.go:122] "RemoveContainer" containerID="e4a980497a93b9958aa774372917a512f8e1df9313e67cad8c209dd6f36d0bd3" Oct 30 22:01:37.881890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.881856 5036 scope.go:122] "RemoveContainer" containerID="e1312d1c01aa21cf9a1c18b98215ebae57a103194b5cef28f3d30dcd87672f1a" Oct 30 22:01:37.906350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:37.906318 5036 scope.go:122] "RemoveContainer" containerID="4e8403af1a25f0384aa381b0a4ba779dd6a604cb91f72cc4d09090672206a330" Oct 30 22:01:38.244383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.244348 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:01:38.245419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.245403 5036 cpu_manager.go:414] "containerMap: removing container" podUID="46b714b5-577a-44c3-9ebb-3ec59b480bed" containerName="revoke-keys" Oct 30 22:01:38.245476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.245423 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b714b5-577a-44c3-9ebb-3ec59b480bed" containerName="revoke-keys" Oct 30 22:01:38.245599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.245588 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b714b5-577a-44c3-9ebb-3ec59b480bed" containerName="revoke-keys" Oct 30 22:01:38.252383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.252358 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.260346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.260317 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:01:38.261704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.261680 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a1ead7\"" Oct 30 22:01:38.274218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.274187 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:01:38.278879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.278854 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.293012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.292979 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:01:38.293863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.293842 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a1ead7-dockercfg-5xxgr\"" Oct 30 22:01:38.395143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.395105 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm8f\" (UniqueName: \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.395503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.395479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.395740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.395726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.396323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.396299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpzv\" (UniqueName: \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.511178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.511097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm8f\" (UniqueName: \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.511464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.511443 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.519468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.519436 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.519756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.519736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpzv\" (UniqueName: \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.520888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.520849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.521527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.521503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.572288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.556028 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm8f\" (UniqueName: \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\") pod \"payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.572288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.558565 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpzv\" (UniqueName: \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\") pod \"payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.595294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.576754 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:38.600575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.600523 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:38.915501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.915477 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:01:38.916016 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:38.915987 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b456ae_17ba_4566_9748_6c7c148879d3.slice/crio-d494e00d82bd32b61a8c286d71beb4ca0736c7b2f01492a720402795327300e7 WatchSource:0}: Error finding container d494e00d82bd32b61a8c286d71beb4ca0736c7b2f01492a720402795327300e7: Status 404 returned error can't find the container with id d494e00d82bd32b61a8c286d71beb4ca0736c7b2f01492a720402795327300e7 Oct 30 22:01:38.950868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:38.950839 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:01:38.958775 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:01:38.958738 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eadb20_bf8b_4a48_acb1_f4409370a990.slice/crio-570ea34f3d522270b31a01ccded53a5faa11d09aaf4b581bb133776bd9092541 WatchSource:0}: Error finding container 570ea34f3d522270b31a01ccded53a5faa11d09aaf4b581bb133776bd9092541: Status 404 returned error can't find the container with id 570ea34f3d522270b31a01ccded53a5faa11d09aaf4b581bb133776bd9092541 Oct 30 22:01:39.130151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.129799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" event={"ID":"b4eadb20-bf8b-4a48-acb1-f4409370a990","Type":"ContainerStarted","Data":"570ea34f3d522270b31a01ccded53a5faa11d09aaf4b581bb133776bd9092541"} Oct 30 22:01:39.132214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.132177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" event={"ID":"95b456ae-17ba-4566-9748-6c7c148879d3","Type":"ContainerStarted","Data":"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba"} Oct 30 22:01:39.132412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.132252 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" event={"ID":"95b456ae-17ba-4566-9748-6c7c148879d3","Type":"ContainerStarted","Data":"d494e00d82bd32b61a8c286d71beb4ca0736c7b2f01492a720402795327300e7"} Oct 30 22:01:39.132529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.132510 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:39.163680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.163574 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" podStartSLOduration=1.163555757 podStartE2EDuration="1.163555757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:39.161143792 +0000 UTC m=+1323.111098751" watchObservedRunningTime="2026-10-30 22:01:39.163555757 +0000 UTC m=+1323.113510704" Oct 30 22:01:39.376218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.376183 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 22:01:39.406616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.403144 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" podUID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" containerName="payload-processing" containerID="cri-o://05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf" gracePeriod=25 Oct 30 22:01:39.438533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.438507 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\") pod \"cc4940a5-b861-40d5-8c0e-6263d1cde106\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " Oct 30 22:01:39.438699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.438680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\" (UniqueName: \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\") pod \"cc4940a5-b861-40d5-8c0e-6263d1cde106\" (UID: \"cc4940a5-b861-40d5-8c0e-6263d1cde106\") " Oct 30 22:01:39.439099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.439072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume" pod "cc4940a5-b861-40d5-8c0e-6263d1cde106" (UID: "cc4940a5-b861-40d5-8c0e-6263d1cde106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:39.439610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.439586 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4940a5-b861-40d5-8c0e-6263d1cde106-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:39.447094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.447062 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7" pod "cc4940a5-b861-40d5-8c0e-6263d1cde106" (UID: "cc4940a5-b861-40d5-8c0e-6263d1cde106"). InnerVolumeSpecName "kube-api-access-kdkv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:39.541679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.541655 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdkv7\" (UniqueName: \"kubernetes.io/projected/cc4940a5-b861-40d5-8c0e-6263d1cde106-kube-api-access-kdkv7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:39.766800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.766775 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:01:39.847025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.846985 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\") pod \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " Oct 30 22:01:39.847203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.847142 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\" (UniqueName: \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\") pod \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\" (UID: \"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b\") " Oct 30 22:01:39.847476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.847452 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume" pod "c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" (UID: "c61ad52d-9c2f-48b9-aeaf-dc50d113b70b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:39.847652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.847635 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:39.849949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.849708 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg" pod "c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" (UID: "c61ad52d-9c2f-48b9-aeaf-dc50d113b70b"). InnerVolumeSpecName "kube-api-access-ldnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:39.949766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:39.949729 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnvg\" (UniqueName: \"kubernetes.io/projected/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b-kube-api-access-ldnvg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:40.014049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.013969 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:01:40.149317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.148997 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" event={"ID":"b4eadb20-bf8b-4a48-acb1-f4409370a990","Type":"ContainerStarted","Data":"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764"} Oct 30 22:01:40.149654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.149625 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:40.152327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.152284 5036 generic.go:356] "Generic (PLEG): container finished" podID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" containerID="05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf" exitCode=0 Oct 30 22:01:40.152525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.152484 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" Oct 30 22:01:40.152589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.152523 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" event={"ID":"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b","Type":"ContainerDied","Data":"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf"} Oct 30 22:01:40.152589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.152570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp" event={"ID":"c61ad52d-9c2f-48b9-aeaf-dc50d113b70b","Type":"ContainerDied","Data":"cc338e38ce96bba3586a4abb2de9f4dc28a1b80e3e4876d7ce6cca233e43e5de"} Oct 30 22:01:40.152710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.152603 5036 scope.go:122] "RemoveContainer" containerID="05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf" Oct 30 22:01:40.154715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.154692 5036 generic.go:356] "Generic (PLEG): container finished" podID="cc4940a5-b861-40d5-8c0e-6263d1cde106" containerID="a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582" exitCode=0 Oct 30 22:01:40.154830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.154739 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" Oct 30 22:01:40.154830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.154775 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" event={"ID":"cc4940a5-b861-40d5-8c0e-6263d1cde106","Type":"ContainerDied","Data":"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582"} Oct 30 22:01:40.154955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.154878 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8" event={"ID":"cc4940a5-b861-40d5-8c0e-6263d1cde106","Type":"ContainerDied","Data":"e5f00b3439875a5a2fe9638430745c1edcbfe3e3800b92c2124e46824bd0fd2d"} Oct 30 22:01:40.173507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.173475 5036 scope.go:122] "RemoveContainer" containerID="05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf" Oct 30 22:01:40.173926 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:40.173888 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf\": container with ID starting with 05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf not found: ID does not exist" containerID="05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf" Oct 30 22:01:40.174040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.173939 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf"} err="failed to get container status \"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf\": rpc error: code = NotFound desc = could not find container \"05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf\": container with ID starting with 05630d72d329d943df32f6e292c1d70d88267e1d31f6dbc72e455c742023f0bf not found: ID does not exist" Oct 30 22:01:40.174040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.173965 5036 scope.go:122] "RemoveContainer" containerID="a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582" Oct 30 22:01:40.174297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.174234 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" podStartSLOduration=2.1742110119999998 podStartE2EDuration="2.174211012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:40.173762891 +0000 UTC m=+1324.123717862" watchObservedRunningTime="2026-10-30 22:01:40.174211012 +0000 UTC m=+1324.124165955" Oct 30 22:01:40.188418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.188393 5036 scope.go:122] "RemoveContainer" containerID="a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582" Oct 30 22:01:40.188747 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:40.188721 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582\": container with ID starting with a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582 not found: ID does not exist" containerID="a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582" Oct 30 22:01:40.188833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.188758 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582"} err="failed to get container status \"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582\": rpc error: code = NotFound desc = could not find container \"a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582\": container with ID starting with a329d54f3b4b7d6d1fa2f60220258ac7d860904bf633ad64bfe074af761cc582 not found: ID does not exist" Oct 30 22:01:40.205199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.205172 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:01:40.211722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.211694 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ecffad-5c6b7c4fc7-qm7xp"] Oct 30 22:01:40.225520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.225500 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 22:01:40.229128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:40.229108 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3654d5-75f5d5865b-hchh8"] Oct 30 22:01:42.050728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:42.050701 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:01:42.195822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:42.195792 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" path="/var/lib/kubelet/pods/c61ad52d-9c2f-48b9-aeaf-dc50d113b70b/volumes" Oct 30 22:01:42.196209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:42.196196 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4940a5-b861-40d5-8c0e-6263d1cde106" path="/var/lib/kubelet/pods/cc4940a5-b861-40d5-8c0e-6263d1cde106/volumes" Oct 30 22:01:43.507407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:43.507368 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm"] Oct 30 22:01:43.511747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:43.511716 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ecffad-c5lbm"] Oct 30 22:01:44.189559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:44.189525 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488ab854-f35c-4872-9f1a-18290dfb9979" path="/var/lib/kubelet/pods/488ab854-f35c-4872-9f1a-18290dfb9979/volumes" Oct 30 22:01:48.100754 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:01:48.100717 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:01:48.591411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:48.591340 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8"] Oct 30 22:01:48.600322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:48.598527 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5289f-cl6b8"] Oct 30 22:01:50.159111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:50.159073 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:01:50.191350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:50.191313 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b714b5-577a-44c3-9ebb-3ec59b480bed" path="/var/lib/kubelet/pods/46b714b5-577a-44c3-9ebb-3ec59b480bed/volumes" Oct 30 22:01:51.170774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:51.170748 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:01:54.003404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.003372 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk"] Oct 30 22:01:54.004091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004075 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cc4940a5-b861-40d5-8c0e-6263d1cde106" containerName="payload-processing" Oct 30 22:01:54.004144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004093 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4940a5-b861-40d5-8c0e-6263d1cde106" containerName="payload-processing" Oct 30 22:01:54.004144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004130 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" containerName="payload-processing" Oct 30 22:01:54.004144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004136 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" containerName="payload-processing" Oct 30 22:01:54.004242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004237 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61ad52d-9c2f-48b9-aeaf-dc50d113b70b" containerName="payload-processing" Oct 30 22:01:54.004321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.004252 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4940a5-b861-40d5-8c0e-6263d1cde106" containerName="payload-processing" Oct 30 22:01:54.029436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.010295 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.056790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.056095 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:01:54.057064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.056669 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:54.057459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.057437 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:54.058600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.058578 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:54.066969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.066901 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk"] Oct 30 22:01:54.081753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.081734 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn"] Oct 30 22:01:54.087091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.087073 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452a6aee-7zdzn"] Oct 30 22:01:54.093457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.093435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlp4p\" (UniqueName: \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.093532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.093470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.093532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.093513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.190124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.190095 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f85b7f8-366d-42fa-9c7a-aad4b96cb58f" path="/var/lib/kubelet/pods/2f85b7f8-366d-42fa-9c7a-aad4b96cb58f/volumes" Oct 30 22:01:54.194203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.194179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlp4p\" (UniqueName: \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.194343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.194215 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.194343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.194255 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.195005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.194933 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.196799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.196779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.206049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.206027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlp4p\" (UniqueName: \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\") pod \"maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.386824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.386799 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:54.759242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:54.759214 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk"] Oct 30 22:01:55.368556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:55.368513 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" event={"ID":"1b782f70-8102-4e43-890b-17a112d458fb","Type":"ContainerStarted","Data":"b4e97a2ef2e5bf5179ee1ea8ddf71b4dba66ff274a73c3982b4c989545b44a04"} Oct 30 22:01:55.369203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:55.369176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" event={"ID":"1b782f70-8102-4e43-890b-17a112d458fb","Type":"ContainerStarted","Data":"3ec342b81b3de6a57e1796647191e60c4ebd781e5ecb7dbcc6b3bf5e9a3b2bf0"} Oct 30 22:01:55.398043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:55.397979 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" podStartSLOduration=2.397960497 podStartE2EDuration="2.397960497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:55.39347548 +0000 UTC m=+1339.343430428" watchObservedRunningTime="2026-10-30 22:01:55.397960497 +0000 UTC m=+1339.347915444" Oct 30 22:01:56.380667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:56.380586 5036 generic.go:356] "Generic (PLEG): container finished" podID="1b782f70-8102-4e43-890b-17a112d458fb" containerID="b4e97a2ef2e5bf5179ee1ea8ddf71b4dba66ff274a73c3982b4c989545b44a04" exitCode=0 Oct 30 22:01:56.381043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:56.380846 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" event={"ID":"1b782f70-8102-4e43-890b-17a112d458fb","Type":"ContainerDied","Data":"b4e97a2ef2e5bf5179ee1ea8ddf71b4dba66ff274a73c3982b4c989545b44a04"} Oct 30 22:01:57.246467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.246438 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 22:01:57.246729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.246704 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" containerID="cri-o://0d654cf0ef3207e7da3375b09e7331ef8a557e9e0e0a344da04e3c057ba4d866" gracePeriod=30 Oct 30 22:01:57.728509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.728470 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:57.804696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.804602 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\" (UniqueName: \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\") pod \"1b782f70-8102-4e43-890b-17a112d458fb\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " Oct 30 22:01:57.804696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.804655 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\") pod \"1b782f70-8102-4e43-890b-17a112d458fb\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " Oct 30 22:01:57.805049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.805022 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\") pod \"1b782f70-8102-4e43-890b-17a112d458fb\" (UID: \"1b782f70-8102-4e43-890b-17a112d458fb\") " Oct 30 22:01:57.805938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.805913 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca" pod "1b782f70-8102-4e43-890b-17a112d458fb" (UID: "1b782f70-8102-4e43-890b-17a112d458fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:57.807596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.807569 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p" pod "1b782f70-8102-4e43-890b-17a112d458fb" (UID: "1b782f70-8102-4e43-890b-17a112d458fb"). InnerVolumeSpecName "kube-api-access-dlp4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:57.808022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.807996 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca" pod "1b782f70-8102-4e43-890b-17a112d458fb" (UID: "1b782f70-8102-4e43-890b-17a112d458fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:57.908554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.908483 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:57.908554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.908517 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlp4p\" (UniqueName: \"kubernetes.io/projected/1b782f70-8102-4e43-890b-17a112d458fb-kube-api-access-dlp4p\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:57.908554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:57.908531 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b782f70-8102-4e43-890b-17a112d458fb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:58.514994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:58.514920 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" Oct 30 22:01:58.514994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:58.514955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk" event={"ID":"1b782f70-8102-4e43-890b-17a112d458fb","Type":"ContainerDied","Data":"3ec342b81b3de6a57e1796647191e60c4ebd781e5ecb7dbcc6b3bf5e9a3b2bf0"} Oct 30 22:01:58.515227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:58.515011 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec342b81b3de6a57e1796647191e60c4ebd781e5ecb7dbcc6b3bf5e9a3b2bf0" Oct 30 22:01:59.695128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.693119 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:01:59.695128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.694415 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1b782f70-8102-4e43-890b-17a112d458fb" containerName="revoke-keys" Oct 30 22:01:59.695128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.694439 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b782f70-8102-4e43-890b-17a112d458fb" containerName="revoke-keys" Oct 30 22:01:59.695128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.694701 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b782f70-8102-4e43-890b-17a112d458fb" containerName="revoke-keys" Oct 30 22:01:59.700611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.700062 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.717171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.717138 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:01:59.724015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.723595 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4cae46-openshift-default-dockercfg-hgg85\"" Oct 30 22:01:59.758640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.758581 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.759235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.759145 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.759622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.759513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.759622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.759578 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.760024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.759966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.760433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.760369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.760768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.760710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.761016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.760909 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9v5b\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.761016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.760981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.868868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.868825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.868940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869024 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869108 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869326 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869383 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869402 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.869972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869945 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9v5b\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.870095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.869994 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.871130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.871024 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.871765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.871739 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.872194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.872174 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.875229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.875199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.885495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.885462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.901439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.900623 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.907070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.907003 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:01:59.907869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:01:59.907838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9v5b\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\") pod \"e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:00.065338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.065231 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:00.371710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.371682 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:02:00.372061 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:00.372042 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc042a5_0e5e_455f_8622_cf662acc6ead.slice/crio-bbda0ce2b5957d2dbc168cbfdc177e11e65b42c628b37d229ebe06b1ac12719d WatchSource:0}: Error finding container bbda0ce2b5957d2dbc168cbfdc177e11e65b42c628b37d229ebe06b1ac12719d: Status 404 returned error can't find the container with id bbda0ce2b5957d2dbc168cbfdc177e11e65b42c628b37d229ebe06b1ac12719d Oct 30 22:02:00.374793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.374755 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:00.374899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.374834 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:00.374899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.374878 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:00.547663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.546822 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerStarted","Data":"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737"} Oct 30 22:02:00.547869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.547706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerStarted","Data":"bbda0ce2b5957d2dbc168cbfdc177e11e65b42c628b37d229ebe06b1ac12719d"} Oct 30 22:02:00.580342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:00.580289 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podStartSLOduration=1.580245021 podStartE2EDuration="1.580245021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:00.57125458 +0000 UTC m=+1344.521209526" watchObservedRunningTime="2026-10-30 22:02:00.580245021 +0000 UTC m=+1344.530200043" Oct 30 22:02:01.066281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:01.066240 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:01.073928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:01.073867 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:01.557707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:01.557661 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:01.558779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:01.558755 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:02.142888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.142854 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:02.147753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.147732 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.164523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.164494 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-71f39e-openshift-default-dockercfg-bzsk6\"" Oct 30 22:02:02.173169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.171748 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:02.256618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.256504 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzmb\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.256773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.256634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.256832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.256812 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.256927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.256909 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.256967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.256945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.257031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.257017 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.257079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.257068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.257165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.257150 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.257218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.257176 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzmb\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359069 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359164 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359218 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359310 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359360 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359447 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.359977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359954 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.360145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.359991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.362094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.362071 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.362255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.362238 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.370820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.370797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzmb\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.371141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.371120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.471128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.468680 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:02.575028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.574997 5036 generic.go:356] "Generic (PLEG): container finished" podID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerID="0d654cf0ef3207e7da3375b09e7331ef8a557e9e0e0a344da04e3c057ba4d866" exitCode=0 Oct 30 22:02:02.575208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.575118 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerDied","Data":"0d654cf0ef3207e7da3375b09e7331ef8a557e9e0e0a344da04e3c057ba4d866"} Oct 30 22:02:02.575208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.575181 5036 scope.go:122] "RemoveContainer" containerID="5689143d0cdb33a710fa4193c5616452bac9087765a96201bd880356e92e78af" Oct 30 22:02:02.577711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.577690 5036 generic.go:356] "Generic (PLEG): container finished" podID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerID="d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082" exitCode=0 Oct 30 22:02:02.577792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.577765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerDied","Data":"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082"} Oct 30 22:02:02.578351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.578328 5036 scope.go:122] "RemoveContainer" containerID="d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082" Oct 30 22:02:02.580119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.580087 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:02.580229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.580161 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:02.580229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.580198 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:02.775842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.775814 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:02.776344 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:02.776316 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf252131e_7606_43f4_a9ec_3e9e69b45e0f.slice/crio-2a29d83d1838b5c548d13d7bbc225f94385956ff7bba4f75f0fb89aee451db54 WatchSource:0}: Error finding container 2a29d83d1838b5c548d13d7bbc225f94385956ff7bba4f75f0fb89aee451db54: Status 404 returned error can't find the container with id 2a29d83d1838b5c548d13d7bbc225f94385956ff7bba4f75f0fb89aee451db54 Oct 30 22:02:02.779213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.779178 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:02.779337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.779252 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:02.779337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:02.779310 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:03.025825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.025438 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:02:03.090189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090156 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090242 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090317 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090387 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090438 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090516 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090639 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.090699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.090678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\") pod \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\" (UID: \"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1\") " Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.093540 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.093773 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.093943 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094016 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094084 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094097 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094289 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094311 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094327 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094342 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094354 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.094394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.094368 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.095396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.095359 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "kube-api-access-hmfj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:03.096433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.096387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:03.098078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.098041 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo" pod "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" (UID: "bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:03.196255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.196206 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.196660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.196277 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfj9\" (UniqueName: \"kubernetes.io/projected/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-kube-api-access-hmfj9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.196660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.196296 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:03.704339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.704244 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" event={"ID":"f252131e-7606-43f4-a9ec-3e9e69b45e0f","Type":"ContainerStarted","Data":"3d307bc3e0f6be8af3fd8b256e7ab735fed6e70360714a80ecd4039fbefbcbd8"} Oct 30 22:02:03.704558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.704429 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" event={"ID":"f252131e-7606-43f4-a9ec-3e9e69b45e0f","Type":"ContainerStarted","Data":"2a29d83d1838b5c548d13d7bbc225f94385956ff7bba4f75f0fb89aee451db54"} Oct 30 22:02:03.706648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.706625 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" Oct 30 22:02:03.706805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.706636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf" event={"ID":"bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1","Type":"ContainerDied","Data":"29dce98df894ddffd21a7e6149445d983955c63504698b67ffff00fb90faef48"} Oct 30 22:02:03.706939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.706924 5036 scope.go:122] "RemoveContainer" containerID="0d654cf0ef3207e7da3375b09e7331ef8a557e9e0e0a344da04e3c057ba4d866" Oct 30 22:02:03.733725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.733687 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerStarted","Data":"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8"} Oct 30 22:02:03.735067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.735018 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podStartSLOduration=1.735000324 podStartE2EDuration="1.735000324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:03.734033728 +0000 UTC m=+1347.683988693" watchObservedRunningTime="2026-10-30 22:02:03.735000324 +0000 UTC m=+1347.684955273" Oct 30 22:02:03.807205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.807170 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 22:02:03.863292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:03.861758 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5289f-openshift-default-77bd85d8dc-5qjcf"] Oct 30 22:02:04.191405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.191372 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" path="/var/lib/kubelet/pods/bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1/volumes" Oct 30 22:02:04.348765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.348727 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:04.349523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.349499 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:04.349720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.349695 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:04.471375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.470893 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:04.472287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.472235 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:04.472465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.472442 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.584875 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9"] Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.586181 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.586205 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.586295 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.586304 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:04.588313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.586498 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:04.624293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.621958 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.655249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.655219 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:04.656150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.656126 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:04.676338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.656793 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:04.676589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.656933 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:02:04.796320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.784092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbql4\" (UniqueName: \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.796320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.784139 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.796320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.784227 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.796815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.796789 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9"] Oct 30 22:02:04.953661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.953620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbql4\" (UniqueName: \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.953919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.953899 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.955331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.955306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.956639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.956614 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.990360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.990327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbql4\" (UniqueName: \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:04.993211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:04.993188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:05.168638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.168594 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:05.356282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.355813 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:05.356282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.355870 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:05.472117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.471633 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:05.472117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.471691 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:05.689943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.689103 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9"] Oct 30 22:02:05.895308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:05.891609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" event={"ID":"de1c2b5b-f459-46c0-b602-1227a37c31e7","Type":"ContainerStarted","Data":"bfa19006d6122ad78150fb5ba04d40bf0a670d3737f8e5a6a882f2a17c26e72a"} Oct 30 22:02:06.484344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.473718 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:06.484344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.473800 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:06.484344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.480477 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:06.484344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.480538 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:06.902615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.902575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" event={"ID":"de1c2b5b-f459-46c0-b602-1227a37c31e7","Type":"ContainerStarted","Data":"0b529d394744529aa69fc45670b6e89bf74b92e0570318bffe483e21bd405da6"} Oct 30 22:02:06.944291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:06.944113 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" podStartSLOduration=2.944093516 podStartE2EDuration="2.944093516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:06.928565591 +0000 UTC m=+1350.878520539" watchObservedRunningTime="2026-10-30 22:02:06.944093516 +0000 UTC m=+1350.894048465" Oct 30 22:02:07.347918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.347824 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:07.348210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.348178 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:07.472492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.472455 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:07.472788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.472757 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:07.938382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.938345 5036 generic.go:356] "Generic (PLEG): container finished" podID="de1c2b5b-f459-46c0-b602-1227a37c31e7" containerID="0b529d394744529aa69fc45670b6e89bf74b92e0570318bffe483e21bd405da6" exitCode=0 Oct 30 22:02:07.939165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:07.939138 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" event={"ID":"de1c2b5b-f459-46c0-b602-1227a37c31e7","Type":"ContainerDied","Data":"0b529d394744529aa69fc45670b6e89bf74b92e0570318bffe483e21bd405da6"} Oct 30 22:02:08.343493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:08.343407 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:08.343754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:08.343728 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:08.473708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:08.473671 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:08.473991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:08.473952 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:09.344491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.344025 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 22:02:09.344491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.344083 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 22:02:09.471957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.471927 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 22:02:09.472191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.472161 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 22:02:09.480838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.480817 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:09.604112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.604016 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\") pod \"de1c2b5b-f459-46c0-b602-1227a37c31e7\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " Oct 30 22:02:09.604390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.604141 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\") pod \"de1c2b5b-f459-46c0-b602-1227a37c31e7\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " Oct 30 22:02:09.605450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.605420 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca" pod "de1c2b5b-f459-46c0-b602-1227a37c31e7" (UID: "de1c2b5b-f459-46c0-b602-1227a37c31e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:09.620324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.620291 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\" (UniqueName: \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\") pod \"de1c2b5b-f459-46c0-b602-1227a37c31e7\" (UID: \"de1c2b5b-f459-46c0-b602-1227a37c31e7\") " Oct 30 22:02:09.622696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.622659 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:09.680755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.680650 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca" pod "de1c2b5b-f459-46c0-b602-1227a37c31e7" (UID: "de1c2b5b-f459-46c0-b602-1227a37c31e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:09.681188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.681166 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4" pod "de1c2b5b-f459-46c0-b602-1227a37c31e7" (UID: "de1c2b5b-f459-46c0-b602-1227a37c31e7"). InnerVolumeSpecName "kube-api-access-jbql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:09.724676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.724518 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbql4\" (UniqueName: \"kubernetes.io/projected/de1c2b5b-f459-46c0-b602-1227a37c31e7-kube-api-access-jbql4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:09.724998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.724866 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1c2b5b-f459-46c0-b602-1227a37c31e7-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:09.867963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:09.867858 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:10.024305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.022433 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" event={"ID":"de1c2b5b-f459-46c0-b602-1227a37c31e7","Type":"ContainerDied","Data":"bfa19006d6122ad78150fb5ba04d40bf0a670d3737f8e5a6a882f2a17c26e72a"} Oct 30 22:02:10.024305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.022588 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa19006d6122ad78150fb5ba04d40bf0a670d3737f8e5a6a882f2a17c26e72a" Oct 30 22:02:10.024305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.022681 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9" Oct 30 22:02:10.345912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.345884 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:10.347537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.347511 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:10.475936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.475907 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:10.476415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.476382 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:10.477222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:10.477203 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:14.772794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:14.772646 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk"] Oct 30 22:02:14.782275 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:14.782227 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3724d9-nlbdk"] Oct 30 22:02:16.251802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:16.251759 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b782f70-8102-4e43-890b-17a112d458fb" path="/var/lib/kubelet/pods/1b782f70-8102-4e43-890b-17a112d458fb/volumes" Oct 30 22:02:18.159804 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:18.159759 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:02:23.654657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:23.654615 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9"] Oct 30 22:02:23.663658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:23.659617 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-71f39e-zcsk9"] Oct 30 22:02:24.189833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:24.189799 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1c2b5b-f459-46c0-b602-1227a37c31e7" path="/var/lib/kubelet/pods/de1c2b5b-f459-46c0-b602-1227a37c31e7/volumes" Oct 30 22:02:25.463097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.463062 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4"] Oct 30 22:02:25.463847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.463824 5036 cpu_manager.go:414] "containerMap: removing container" podUID="de1c2b5b-f459-46c0-b602-1227a37c31e7" containerName="revoke-keys" Oct 30 22:02:25.463847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.463846 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1c2b5b-f459-46c0-b602-1227a37c31e7" containerName="revoke-keys" Oct 30 22:02:25.464049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.463983 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeb3a42-61b8-48b3-813d-4ee3fc4cedb1" containerName="istio-proxy" Oct 30 22:02:25.464049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.463995 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1c2b5b-f459-46c0-b602-1227a37c31e7" containerName="revoke-keys" Oct 30 22:02:25.467998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.467981 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.472157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.472135 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:25.474500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.473444 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:25.474867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.474832 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:25.474976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.474945 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:02:25.478981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.478938 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4"] Oct 30 22:02:25.546081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.546038 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.546289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.546109 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.546535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.546516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltskg\" (UniqueName: \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.648066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.648032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltskg\" (UniqueName: \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.648277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.648100 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.648277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.648138 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.649125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.649097 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.650809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.650779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.660147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.660126 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltskg\" (UniqueName: \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\") pod \"maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:25.787668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:25.787591 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:26.104000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.103959 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4"] Oct 30 22:02:26.104711 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:26.104686 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa89f84_3fa0_477b_ab73_1d3009853b36.slice/crio-dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd WatchSource:0}: Error finding container dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd: Status 404 returned error can't find the container with id dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd Oct 30 22:02:26.220504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.220468 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh"] Oct 30 22:02:26.226968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.226944 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.240660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.239239 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh"] Oct 30 22:02:26.260222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.260185 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.260405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.260376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrf7v\" (UniqueName: \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.260528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.260510 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.362566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.362469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrf7v\" (UniqueName: \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.362768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.362594 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.362768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.362736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.363283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.363243 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.365346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.365329 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.373972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.373954 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrf7v\" (UniqueName: \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\") pod \"maas-api-revoke-sub-dd25d820-7dvgh\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.422653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.422616 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" event={"ID":"bfa89f84-3fa0-477b-ab73-1d3009853b36","Type":"ContainerStarted","Data":"dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd"} Oct 30 22:02:26.546873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.546838 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:26.931399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.919388 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 22:02:26.931399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:26.919647 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" containerID="cri-o://368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b" gracePeriod=30 Oct 30 22:02:27.160846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:27.153669 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh"] Oct 30 22:02:27.442601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:27.442210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" event={"ID":"61fe99f9-983f-4adc-877d-e8c1ac60250e","Type":"ContainerStarted","Data":"affb2b00aad32b782429f1d694fb2678df166c10a3a06515080158b63d863732"} Oct 30 22:02:27.444253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:27.444222 5036 generic.go:356] "Generic (PLEG): container finished" podID="bfa89f84-3fa0-477b-ab73-1d3009853b36" containerID="00e0fe4cc506a744e3fde98ca6148bbca91ef624f5d6fb0cd6d962088a558645" exitCode=0 Oct 30 22:02:27.444428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:27.444348 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" event={"ID":"bfa89f84-3fa0-477b-ab73-1d3009853b36","Type":"ContainerDied","Data":"00e0fe4cc506a744e3fde98ca6148bbca91ef624f5d6fb0cd6d962088a558645"} Oct 30 22:02:28.460970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.460933 5036 generic.go:356] "Generic (PLEG): container finished" podID="61fe99f9-983f-4adc-877d-e8c1ac60250e" containerID="51f1fd278b3876a4f23890c00498cf1ac78720030631d08803bd732832b8778d" exitCode=0 Oct 30 22:02:28.461493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.461026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" event={"ID":"61fe99f9-983f-4adc-877d-e8c1ac60250e","Type":"ContainerDied","Data":"51f1fd278b3876a4f23890c00498cf1ac78720030631d08803bd732832b8778d"} Oct 30 22:02:28.463223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.463200 5036 generic.go:356] "Generic (PLEG): container finished" podID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerID="f70b851585945a87dbe6f4e661b13826a02c64c7ced1f255dfac15ee6acc925a" exitCode=0 Oct 30 22:02:28.463362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.463302 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerDied","Data":"f70b851585945a87dbe6f4e661b13826a02c64c7ced1f255dfac15ee6acc925a"} Oct 30 22:02:28.464167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.463941 5036 scope.go:122] "RemoveContainer" containerID="f70b851585945a87dbe6f4e661b13826a02c64c7ced1f255dfac15ee6acc925a" Oct 30 22:02:28.465576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.465545 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:28.465696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.465614 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:28.465696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.465673 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:28.714985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.714954 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:28.769416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.769387 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\" (UniqueName: \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\") pod \"bfa89f84-3fa0-477b-ab73-1d3009853b36\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " Oct 30 22:02:28.769615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.769465 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\") pod \"bfa89f84-3fa0-477b-ab73-1d3009853b36\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " Oct 30 22:02:28.769615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.769583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\") pod \"bfa89f84-3fa0-477b-ab73-1d3009853b36\" (UID: \"bfa89f84-3fa0-477b-ab73-1d3009853b36\") " Oct 30 22:02:28.770235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.770195 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca" pod "bfa89f84-3fa0-477b-ab73-1d3009853b36" (UID: "bfa89f84-3fa0-477b-ab73-1d3009853b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:28.772104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.772071 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca" pod "bfa89f84-3fa0-477b-ab73-1d3009853b36" (UID: "bfa89f84-3fa0-477b-ab73-1d3009853b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:28.773229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.773204 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg" pod "bfa89f84-3fa0-477b-ab73-1d3009853b36" (UID: "bfa89f84-3fa0-477b-ab73-1d3009853b36"). InnerVolumeSpecName "kube-api-access-ltskg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:28.871322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.871288 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.871322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.871319 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltskg\" (UniqueName: \"kubernetes.io/projected/bfa89f84-3fa0-477b-ab73-1d3009853b36-kube-api-access-ltskg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:28.871322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:28.871329 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa89f84-3fa0-477b-ab73-1d3009853b36-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.474255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.474171 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" Oct 30 22:02:29.474255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.474204 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4" event={"ID":"bfa89f84-3fa0-477b-ab73-1d3009853b36","Type":"ContainerDied","Data":"dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd"} Oct 30 22:02:29.474255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.474253 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee7a28d399b00914bb6be5a2846e7334d854cc940cbdc21076386e4da2e0dfd" Oct 30 22:02:29.476666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.476639 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerStarted","Data":"ea0bc7207715166a9cdb6445d370dbc8bd8b1278e27c9b75e5dfc8f6e56da258"} Oct 30 22:02:29.759796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.759563 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:02:29.760211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.760167 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.760350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.760208 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:29.794732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.794705 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:29.890528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.890494 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\") pod \"61fe99f9-983f-4adc-877d-e8c1ac60250e\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " Oct 30 22:02:29.890752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.890627 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\") pod \"61fe99f9-983f-4adc-877d-e8c1ac60250e\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " Oct 30 22:02:29.890752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.890704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\" (UniqueName: \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\") pod \"61fe99f9-983f-4adc-877d-e8c1ac60250e\" (UID: \"61fe99f9-983f-4adc-877d-e8c1ac60250e\") " Oct 30 22:02:29.890886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.890864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca" pod "61fe99f9-983f-4adc-877d-e8c1ac60250e" (UID: "61fe99f9-983f-4adc-877d-e8c1ac60250e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:29.891466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.891442 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.893018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.892991 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca" pod "61fe99f9-983f-4adc-877d-e8c1ac60250e" (UID: "61fe99f9-983f-4adc-877d-e8c1ac60250e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:29.893118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.893069 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v" pod "61fe99f9-983f-4adc-877d-e8c1ac60250e" (UID: "61fe99f9-983f-4adc-877d-e8c1ac60250e"). InnerVolumeSpecName "kube-api-access-wrf7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:29.993150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.993116 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrf7v\" (UniqueName: \"kubernetes.io/projected/61fe99f9-983f-4adc-877d-e8c1ac60250e-kube-api-access-wrf7v\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.993150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:29.993150 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fe99f9-983f-4adc-877d-e8c1ac60250e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:30.499429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:30.499342 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" Oct 30 22:02:30.503477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:30.503324 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh" event={"ID":"61fe99f9-983f-4adc-877d-e8c1ac60250e","Type":"ContainerDied","Data":"affb2b00aad32b782429f1d694fb2678df166c10a3a06515080158b63d863732"} Oct 30 22:02:30.503477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:30.503386 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="affb2b00aad32b782429f1d694fb2678df166c10a3a06515080158b63d863732" Oct 30 22:02:30.759985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:30.759887 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.759985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:30.759942 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:31.764371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:31.764325 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.764964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:31.764929 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:32.390238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.390206 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 22:02:32.479025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.478979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.479433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.479407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.479694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.479674 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.479912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.479897 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.501293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.488255 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.513434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.513518 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.513550 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.513591 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\") pod \"47de940a-79cf-418a-8983-35541566b94a\" (UID: \"47de940a-79cf-418a-8983-35541566b94a\") " Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.496234 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.512752 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.513028 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.516640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.516678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.569500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.550772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:32.578288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.574282 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:32.601871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.587697 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47de940a-79cf-418a-8983-35541566b94a-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.601871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.587732 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.601871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.587746 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.603627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.603396 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" containerID="cri-o://3d307bc3e0f6be8af3fd8b256e7ab735fed6e70360714a80ecd4039fbefbcbd8" gracePeriod=30 Oct 30 22:02:32.683312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.653872 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "kube-api-access-kx9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.701402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.692367 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47de940a-79cf-418a-8983-35541566b94a-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.701402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.692406 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9cx\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-kube-api-access-kx9cx\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.701402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.692422 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.701402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.692437 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.790346 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q"] Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791530 5036 cpu_manager.go:414] "containerMap: removing container" podUID="61fe99f9-983f-4adc-877d-e8c1ac60250e" containerName="revoke-keys" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791551 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe99f9-983f-4adc-877d-e8c1ac60250e" containerName="revoke-keys" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791593 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bfa89f84-3fa0-477b-ab73-1d3009853b36" containerName="revoke-keys" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791601 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa89f84-3fa0-477b-ab73-1d3009853b36" containerName="revoke-keys" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791662 5036 cpu_manager.go:414] "containerMap: removing container" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791669 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791819 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="47de940a-79cf-418a-8983-35541566b94a" containerName="istio-proxy" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791834 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe99f9-983f-4adc-877d-e8c1ac60250e" containerName="revoke-keys" Oct 30 22:02:32.793889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.791849 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa89f84-3fa0-477b-ab73-1d3009853b36" containerName="revoke-keys" Oct 30 22:02:32.796301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.794653 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:32.796301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.794711 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.797568 5036 generic.go:356] "Generic (PLEG): container finished" podID="47de940a-79cf-418a-8983-35541566b94a" containerID="368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b" exitCode=0 Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.797925 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.798143 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" event={"ID":"47de940a-79cf-418a-8983-35541566b94a","Type":"ContainerDied","Data":"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b"} Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.798179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf" event={"ID":"47de940a-79cf-418a-8983-35541566b94a","Type":"ContainerDied","Data":"81d445bc2711bfb1292fd9f2d9e76b55b5ca9d7a9927c31d1dfc1168905ea179"} Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.798293 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:32.798953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.798573 5036 scope.go:122] "RemoveContainer" containerID="368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b" Oct 30 22:02:32.809786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.805364 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:32.809786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.805529 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:02:32.809786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.805559 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:32.826337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.813833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.862394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.844948 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q"] Oct 30 22:02:32.888751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.888595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy" pod "47de940a-79cf-418a-8983-35541566b94a" (UID: "47de940a-79cf-418a-8983-35541566b94a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.903630 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.903691 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.903779 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smbcq\" (UniqueName: \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.904099 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47de940a-79cf-418a-8983-35541566b94a-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.904115 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47de940a-79cf-418a-8983-35541566b94a-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.952042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:32.907524 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:33.098847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.098490 5036 scope.go:122] "RemoveContainer" containerID="368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b" Oct 30 22:02:33.099240 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:33.099133 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b\": container with ID starting with 368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b not found: ID does not exist" containerID="368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b" Oct 30 22:02:33.099240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.099178 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b"} err="failed to get container status \"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b\": rpc error: code = NotFound desc = could not find container \"368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b\": container with ID starting with 368ed68adf6bdb513137f3186d544ca28eb610e61c302d4d3391e568978a455b not found: ID does not exist" Oct 30 22:02:33.105845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.105219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.105845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.105282 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.105845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.105340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smbcq\" (UniqueName: \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.118942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.118882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.149978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.135023 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.167524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.161670 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt"] Oct 30 22:02:33.171402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.171298 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smbcq\" (UniqueName: \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.173695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.173671 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt"] Oct 30 22:02:33.173803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.173740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.192618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.192594 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:33.206659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.206627 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.206858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.206839 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.206957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.206942 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxk42\" (UniqueName: \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.207893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.207872 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 22:02:33.222319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.214505 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-d6sgf"] Oct 30 22:02:33.310214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.310181 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.310214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.310428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.310214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.310497 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxk42\" (UniqueName: \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.347489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.341027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.359347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.357144 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.359347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.358731 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxk42\" (UniqueName: \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\") pod \"maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.502853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.502818 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:33.582197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.582159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q"] Oct 30 22:02:33.585066 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:33.585041 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c655c7_9d5e_482d_8225_d2c0fdb41f8e.slice/crio-d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8 WatchSource:0}: Error finding container d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8: Status 404 returned error can't find the container with id d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8 Oct 30 22:02:33.770045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.769956 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:33.770328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.770297 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:33.866289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:33.865970 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" event={"ID":"01c655c7-9d5e-482d-8225-d2c0fdb41f8e","Type":"ContainerStarted","Data":"d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8"} Oct 30 22:02:34.198680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.198644 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47de940a-79cf-418a-8983-35541566b94a" path="/var/lib/kubelet/pods/47de940a-79cf-418a-8983-35541566b94a/volumes" Oct 30 22:02:34.383217 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:34.383074 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbb1c13_79f0_47c4_80cd_32a1b76f6ef9.slice/crio-cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39 WatchSource:0}: Error finding container cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39: Status 404 returned error can't find the container with id cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39 Oct 30 22:02:34.384494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.384468 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt"] Oct 30 22:02:34.763290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.763116 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:34.763290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.763188 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:34.899393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.899319 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" event={"ID":"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9","Type":"ContainerStarted","Data":"cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39"} Oct 30 22:02:34.902137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.902101 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" event={"ID":"01c655c7-9d5e-482d-8225-d2c0fdb41f8e","Type":"ContainerStarted","Data":"6889a0e914efd708c11f939ba191009223ffb9018982781c0201b0d8c72b05e4"} Oct 30 22:02:34.983955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:34.983883 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" podStartSLOduration=2.983861736 podStartE2EDuration="2.983861736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:34.976798872 +0000 UTC m=+1378.926753815" watchObservedRunningTime="2026-10-30 22:02:34.983861736 +0000 UTC m=+1378.933816684" Oct 30 22:02:35.090516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.090474 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:02:35.760294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.760068 5036 patch_prober.go:29] interesting pod/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:02:35.760294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.760146 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 22:02:35.956631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.956595 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:02:35.969639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.962942 5036 generic.go:356] "Generic (PLEG): container finished" podID="01c655c7-9d5e-482d-8225-d2c0fdb41f8e" containerID="6889a0e914efd708c11f939ba191009223ffb9018982781c0201b0d8c72b05e4" exitCode=0 Oct 30 22:02:35.969639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.963184 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" event={"ID":"01c655c7-9d5e-482d-8225-d2c0fdb41f8e","Type":"ContainerDied","Data":"6889a0e914efd708c11f939ba191009223ffb9018982781c0201b0d8c72b05e4"} Oct 30 22:02:35.969639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.969391 5036 generic.go:356] "Generic (PLEG): container finished" podID="0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" containerID="12ab4d3e6f5702ed3253f6c6c2000de795b603ac784592ec8ed48412e3222c06" exitCode=0 Oct 30 22:02:35.969639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.969571 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" event={"ID":"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9","Type":"ContainerDied","Data":"12ab4d3e6f5702ed3253f6c6c2000de795b603ac784592ec8ed48412e3222c06"} Oct 30 22:02:35.984642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:35.976804 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:02:36.763798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:36.763768 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:02:36.765113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:36.765091 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:02:37.474428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.474400 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:37.476484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.476465 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:37.606795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.606768 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\") pod \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " Oct 30 22:02:37.606969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.606821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\") pod \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " Oct 30 22:02:37.606969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.606949 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\") pod \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " Oct 30 22:02:37.607065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607051 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\") pod \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " Oct 30 22:02:37.607118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\" (UniqueName: \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\") pod \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\" (UID: \"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9\") " Oct 30 22:02:37.607118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607101 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca" pod "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" (UID: "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:37.607318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607292 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\" (UniqueName: \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\") pod \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\" (UID: \"01c655c7-9d5e-482d-8225-d2c0fdb41f8e\") " Oct 30 22:02:37.607421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607350 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca" pod "01c655c7-9d5e-482d-8225-d2c0fdb41f8e" (UID: "01c655c7-9d5e-482d-8225-d2c0fdb41f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:37.607906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607886 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.607906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.607907 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.609506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.609481 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca" pod "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" (UID: "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:37.609593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.609574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42" pod "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" (UID: "0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9"). InnerVolumeSpecName "kube-api-access-pxk42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:37.609727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.609701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca" pod "01c655c7-9d5e-482d-8225-d2c0fdb41f8e" (UID: "01c655c7-9d5e-482d-8225-d2c0fdb41f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:37.609941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.609912 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq" pod "01c655c7-9d5e-482d-8225-d2c0fdb41f8e" (UID: "01c655c7-9d5e-482d-8225-d2c0fdb41f8e"). InnerVolumeSpecName "kube-api-access-smbcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:37.709456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.709429 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smbcq\" (UniqueName: \"kubernetes.io/projected/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-kube-api-access-smbcq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.709456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.709454 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c655c7-9d5e-482d-8225-d2c0fdb41f8e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.709649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.709465 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.709649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:37.709475 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxk42\" (UniqueName: \"kubernetes.io/projected/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9-kube-api-access-pxk42\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.091840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.087341 5036 scope.go:122] "RemoveContainer" containerID="734a8844da6c6b2d3ec003ef66e5cdc315993a67f507314d223d0a5840a58497" Oct 30 22:02:38.104639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.104611 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" Oct 30 22:02:38.104836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.104637 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt" event={"ID":"0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9","Type":"ContainerDied","Data":"cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39"} Oct 30 22:02:38.104836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.104719 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39" Oct 30 22:02:38.158557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.150308 5036 generic.go:356] "Generic (PLEG): container finished" podID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerID="3d307bc3e0f6be8af3fd8b256e7ab735fed6e70360714a80ecd4039fbefbcbd8" exitCode=0 Oct 30 22:02:38.158557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.150533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" event={"ID":"f252131e-7606-43f4-a9ec-3e9e69b45e0f","Type":"ContainerDied","Data":"3d307bc3e0f6be8af3fd8b256e7ab735fed6e70360714a80ecd4039fbefbcbd8"} Oct 30 22:02:38.158557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.154416 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" event={"ID":"01c655c7-9d5e-482d-8225-d2c0fdb41f8e","Type":"ContainerDied","Data":"d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8"} Oct 30 22:02:38.158557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.154436 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q" Oct 30 22:02:38.158557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.154542 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d7ed8ebb27c5a9b358a2fe16ca641bdffcbbd11cd065871b0d0b713df455e8" Oct 30 22:02:38.187829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.187807 5036 scope.go:122] "RemoveContainer" containerID="469cd67a34b1d5cefe47ed2195abd46b61c6b5029cbbe1a8dca60be06c92ec47" Oct 30 22:02:38.259467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.259439 5036 scope.go:122] "RemoveContainer" containerID="fc401cb428ec5b285a53f7104f2a20e8270a5e335c0ab2c6dba724db24fee09c" Oct 30 22:02:38.279178 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:38.279135 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbb1c13_79f0_47c4_80cd_32a1b76f6ef9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbb1c13_79f0_47c4_80cd_32a1b76f6ef9.slice/crio-cc5532e77fcc04c4f6d03e52efb3d1f658223227c0ea9b75ae2cc291dc759d39\": RecentStats: unable to find data in memory cache]" Oct 30 22:02:38.284055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.283940 5036 scope.go:122] "RemoveContainer" containerID="33a3c1763bb5d7460948c9db94db527f32d2eb37cb77e597b222656ca94f18bc" Oct 30 22:02:38.391139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.390826 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:38.478015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.477992 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478072 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478119 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478410 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478510 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478613 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478635 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478657 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478710 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\") pod \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\" (UID: \"f252131e-7606-43f4-a9ec-3e9e69b45e0f\") " Oct 30 22:02:38.478942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478846 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.479298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.478998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.479298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479019 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.479434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479344 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.479434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479361 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.479434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479370 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.479434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479379 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.479434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.479393 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.480692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.480669 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.480947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.480929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:38.481232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.481215 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "kube-api-access-hvzmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.481872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.481853 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy" pod "f252131e-7606-43f4-a9ec-3e9e69b45e0f" (UID: "f252131e-7606-43f4-a9ec-3e9e69b45e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.580379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.580288 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.580379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.580326 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzmb\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-kube-api-access-hvzmb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.580379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.580336 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.580379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:38.580353 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f252131e-7606-43f4-a9ec-3e9e69b45e0f-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:39.166117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.166074 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" event={"ID":"f252131e-7606-43f4-a9ec-3e9e69b45e0f","Type":"ContainerDied","Data":"2a29d83d1838b5c548d13d7bbc225f94385956ff7bba4f75f0fb89aee451db54"} Oct 30 22:02:39.166347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.166148 5036 scope.go:122] "RemoveContainer" containerID="3d307bc3e0f6be8af3fd8b256e7ab735fed6e70360714a80ecd4039fbefbcbd8" Oct 30 22:02:39.166347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.166090 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr" Oct 30 22:02:39.193088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.193050 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:39.199550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.199525 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-4thvr"] Oct 30 22:02:39.203679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.203657 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8"] Oct 30 22:02:39.204374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204359 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" Oct 30 22:02:39.204416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204375 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" Oct 30 22:02:39.204416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204412 5036 cpu_manager.go:414] "containerMap: removing container" podUID="01c655c7-9d5e-482d-8225-d2c0fdb41f8e" containerName="revoke-keys" Oct 30 22:02:39.204416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204416 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c655c7-9d5e-482d-8225-d2c0fdb41f8e" containerName="revoke-keys" Oct 30 22:02:39.204517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204435 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" containerName="revoke-keys" Oct 30 22:02:39.204517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204441 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" containerName="revoke-keys" Oct 30 22:02:39.204577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204552 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" containerName="revoke-keys" Oct 30 22:02:39.204577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204564 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" containerName="istio-proxy" Oct 30 22:02:39.204577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.204576 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c655c7-9d5e-482d-8225-d2c0fdb41f8e" containerName="revoke-keys" Oct 30 22:02:39.210136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.210102 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.224594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.224563 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:39.225369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.225347 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:39.225642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.225594 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:39.239315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.238156 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:02:39.239998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.239880 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8"] Oct 30 22:02:39.292116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.292079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.292287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.292188 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.292287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.292220 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjn7\" (UniqueName: \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.394224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.394198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.394383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.394241 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.394383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.394285 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjn7\" (UniqueName: \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.394914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.394892 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.419383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.397226 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.430971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.430845 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjn7\" (UniqueName: \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\") pod \"maas-api-revoke-sub-9336f746-d8hr8\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.550230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.550190 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:39.848972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:39.848941 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8"] Oct 30 22:02:39.849896 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:39.849870 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e10802_7f82_4a19_992f_9668811d03dd.slice/crio-43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730 WatchSource:0}: Error finding container 43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730: Status 404 returned error can't find the container with id 43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730 Oct 30 22:02:40.196410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:40.196373 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f252131e-7606-43f4-a9ec-3e9e69b45e0f" path="/var/lib/kubelet/pods/f252131e-7606-43f4-a9ec-3e9e69b45e0f/volumes" Oct 30 22:02:40.196892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:40.196866 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" event={"ID":"b1e10802-7f82-4a19-992f-9668811d03dd","Type":"ContainerStarted","Data":"43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730"} Oct 30 22:02:40.957898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:40.957678 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" podUID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" containerName="payload-processing" containerID="cri-o://7518dad7e7192d780fd4777e3b4112a2c31bae9e82c61d09b9ef7e1b8eb01ec2" gracePeriod=25 Oct 30 22:02:40.978057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:40.978016 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" podUID="ded3a111-52af-4ca7-beea-2779769345f6" containerName="payload-pre-processing" containerID="cri-o://3e1ab81c1b8cc15549ee4a63ac63696f95e151cbcb72db89badbfc25652f9bfd" gracePeriod=25 Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.231236 5036 generic.go:356] "Generic (PLEG): container finished" podID="b1e10802-7f82-4a19-992f-9668811d03dd" containerID="69756596d30707c7b14b51d1efe64e6b7264a1385a30849657032dfb31cfbdf6" exitCode=0 Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.231464 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" event={"ID":"b1e10802-7f82-4a19-992f-9668811d03dd","Type":"ContainerDied","Data":"69756596d30707c7b14b51d1efe64e6b7264a1385a30849657032dfb31cfbdf6"} Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.234142 5036 generic.go:356] "Generic (PLEG): container finished" podID="ded3a111-52af-4ca7-beea-2779769345f6" containerID="3e1ab81c1b8cc15549ee4a63ac63696f95e151cbcb72db89badbfc25652f9bfd" exitCode=0 Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.234212 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" event={"ID":"ded3a111-52af-4ca7-beea-2779769345f6","Type":"ContainerDied","Data":"3e1ab81c1b8cc15549ee4a63ac63696f95e151cbcb72db89badbfc25652f9bfd"} Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.236373 5036 generic.go:356] "Generic (PLEG): container finished" podID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" containerID="7518dad7e7192d780fd4777e3b4112a2c31bae9e82c61d09b9ef7e1b8eb01ec2" exitCode=0 Oct 30 22:02:41.250327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.236443 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" event={"ID":"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb","Type":"ContainerDied","Data":"7518dad7e7192d780fd4777e3b4112a2c31bae9e82c61d09b9ef7e1b8eb01ec2"} Oct 30 22:02:41.306303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.301146 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:02:41.362820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.362796 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.445360 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\" (UniqueName: \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\") pod \"ded3a111-52af-4ca7-beea-2779769345f6\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.445449 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\") pod \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.445483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\" (UniqueName: \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\") pod \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\" (UID: \"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb\") " Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.445644 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\") pod \"ded3a111-52af-4ca7-beea-2779769345f6\" (UID: \"ded3a111-52af-4ca7-beea-2779769345f6\") " Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.446654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume" pod "ded3a111-52af-4ca7-beea-2779769345f6" (UID: "ded3a111-52af-4ca7-beea-2779769345f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.447699 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume" pod "ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" (UID: "ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.449088 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl" pod "ded3a111-52af-4ca7-beea-2779769345f6" (UID: "ded3a111-52af-4ca7-beea-2779769345f6"). InnerVolumeSpecName "kube-api-access-25xxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:41.451426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.450798 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9" pod "ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" (UID: "ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb"). InnerVolumeSpecName "kube-api-access-x5mp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:41.547958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.547922 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3a111-52af-4ca7-beea-2779769345f6-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.547958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.547957 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25xxl\" (UniqueName: \"kubernetes.io/projected/ded3a111-52af-4ca7-beea-2779769345f6-kube-api-access-25xxl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.547958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.547966 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.548209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:41.547975 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5mp9\" (UniqueName: \"kubernetes.io/projected/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb-kube-api-access-x5mp9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:42.250880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.250845 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" Oct 30 22:02:42.251398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.250846 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2" event={"ID":"ded3a111-52af-4ca7-beea-2779769345f6","Type":"ContainerDied","Data":"5b8f1eb70c53e7aed12f6d444854d13bad0f3902692d367ac9ac6ae66e473b5d"} Oct 30 22:02:42.251398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.251015 5036 scope.go:122] "RemoveContainer" containerID="3e1ab81c1b8cc15549ee4a63ac63696f95e151cbcb72db89badbfc25652f9bfd" Oct 30 22:02:42.253584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.252786 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" Oct 30 22:02:42.253584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.252824 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7" event={"ID":"ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb","Type":"ContainerDied","Data":"8f213726de783a9c763c677f2a86de6dddfba76d6e1fb8df3f3b9c2b54977d9a"} Oct 30 22:02:42.277762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.277685 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:02:42.282574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.282435 5036 scope.go:122] "RemoveContainer" containerID="7518dad7e7192d780fd4777e3b4112a2c31bae9e82c61d09b9ef7e1b8eb01ec2" Oct 30 22:02:42.284729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.284708 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c097ae-788d74f6fc-g6pd7"] Oct 30 22:02:42.294660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.294637 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:02:42.303076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.302681 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c097ae-55f4cf4d87-6f8n2"] Oct 30 22:02:42.506943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.506907 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:42.575478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.575442 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\") pod \"b1e10802-7f82-4a19-992f-9668811d03dd\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " Oct 30 22:02:42.575699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.575662 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\") pod \"b1e10802-7f82-4a19-992f-9668811d03dd\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " Oct 30 22:02:42.575808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.575755 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\" (UniqueName: \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\") pod \"b1e10802-7f82-4a19-992f-9668811d03dd\" (UID: \"b1e10802-7f82-4a19-992f-9668811d03dd\") " Oct 30 22:02:42.575808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.575793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca" pod "b1e10802-7f82-4a19-992f-9668811d03dd" (UID: "b1e10802-7f82-4a19-992f-9668811d03dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:42.576431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.576404 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:42.578361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.578339 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca" pod "b1e10802-7f82-4a19-992f-9668811d03dd" (UID: "b1e10802-7f82-4a19-992f-9668811d03dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:42.578984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.578960 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7" pod "b1e10802-7f82-4a19-992f-9668811d03dd" (UID: "b1e10802-7f82-4a19-992f-9668811d03dd"). InnerVolumeSpecName "kube-api-access-6sjn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:42.677716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.677682 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e10802-7f82-4a19-992f-9668811d03dd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:42.677716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:42.677713 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjn7\" (UniqueName: \"kubernetes.io/projected/b1e10802-7f82-4a19-992f-9668811d03dd-kube-api-access-6sjn7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:43.362209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.362167 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" event={"ID":"b1e10802-7f82-4a19-992f-9668811d03dd","Type":"ContainerDied","Data":"43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730"} Oct 30 22:02:43.362735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.362238 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43381016323b4d961520698dc6f701cdfabc374b8e138a108019e06e9bc0e730" Oct 30 22:02:43.362735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.362356 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8" Oct 30 22:02:43.386669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.386629 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4"] Oct 30 22:02:43.394277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.394222 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c097ae-vm6m4"] Oct 30 22:02:43.573809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.573768 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:02:43.586170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:43.586133 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:02:44.194197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.194149 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa89f84-3fa0-477b-ab73-1d3009853b36" path="/var/lib/kubelet/pods/bfa89f84-3fa0-477b-ab73-1d3009853b36/volumes" Oct 30 22:02:44.196700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.194821 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded3a111-52af-4ca7-beea-2779769345f6" path="/var/lib/kubelet/pods/ded3a111-52af-4ca7-beea-2779769345f6/volumes" Oct 30 22:02:44.196700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.195394 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" path="/var/lib/kubelet/pods/ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb/volumes" Oct 30 22:02:44.394404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.394375 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerID="6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737" exitCode=0 Oct 30 22:02:44.394906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.394449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerDied","Data":"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737"} Oct 30 22:02:44.395020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.395000 5036 scope.go:122] "RemoveContainer" containerID="6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737" Oct 30 22:02:44.396643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.396616 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:44.396747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.396682 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:44.396747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:44.396715 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:45.407424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:45.407389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerStarted","Data":"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0"} Oct 30 22:02:46.065745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.065716 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:46.067062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.067036 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:46.067198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.067091 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:46.390914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.390878 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 22:02:46.391769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.391128 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" podUID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" containerName="payload-processing" containerID="cri-o://5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715" gracePeriod=30 Oct 30 22:02:46.405856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.405814 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 22:02:46.406179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.406115 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" podUID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" containerName="payload-pre-processing" containerID="cri-o://396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311" gracePeriod=30 Oct 30 22:02:46.560160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.560126 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:46.884287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.881417 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q"] Oct 30 22:02:46.886869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:46.886822 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e3670-qx68q"] Oct 30 22:02:47.070088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.070056 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:47.070366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.070189 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:47.281972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.281882 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 22:02:47.351515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.351479 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 22:02:47.403686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.403656 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\") pod \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " Oct 30 22:02:47.403884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.403761 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\") pod \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " Oct 30 22:02:47.403884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.403796 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\" (UniqueName: \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\") pod \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\" (UID: \"52b962ac-f06e-4e57-92a5-74248dcd4eb1\") " Oct 30 22:02:47.404028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.404012 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\" (UniqueName: \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\") pod \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\" (UID: \"23fb2247-fb37-4e64-81df-47ffaa1c38b5\") " Oct 30 22:02:47.404314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.404281 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume" pod "52b962ac-f06e-4e57-92a5-74248dcd4eb1" (UID: "52b962ac-f06e-4e57-92a5-74248dcd4eb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:47.404490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.404469 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume" pod "23fb2247-fb37-4e64-81df-47ffaa1c38b5" (UID: "23fb2247-fb37-4e64-81df-47ffaa1c38b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:47.404667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.404647 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fb2247-fb37-4e64-81df-47ffaa1c38b5-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.404749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.404676 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b962ac-f06e-4e57-92a5-74248dcd4eb1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.406690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.406660 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq" pod "52b962ac-f06e-4e57-92a5-74248dcd4eb1" (UID: "52b962ac-f06e-4e57-92a5-74248dcd4eb1"). InnerVolumeSpecName "kube-api-access-2mjdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:47.407116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.407095 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27" pod "23fb2247-fb37-4e64-81df-47ffaa1c38b5" (UID: "23fb2247-fb37-4e64-81df-47ffaa1c38b5"). InnerVolumeSpecName "kube-api-access-bxt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:47.506239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.506149 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt27\" (UniqueName: \"kubernetes.io/projected/23fb2247-fb37-4e64-81df-47ffaa1c38b5-kube-api-access-bxt27\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.506239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.506185 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjdq\" (UniqueName: \"kubernetes.io/projected/52b962ac-f06e-4e57-92a5-74248dcd4eb1-kube-api-access-2mjdq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.550127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.550079 5036 generic.go:356] "Generic (PLEG): container finished" podID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" containerID="396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311" exitCode=0 Oct 30 22:02:47.550401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.550355 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" event={"ID":"23fb2247-fb37-4e64-81df-47ffaa1c38b5","Type":"ContainerDied","Data":"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311"} Oct 30 22:02:47.550498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.550419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" event={"ID":"23fb2247-fb37-4e64-81df-47ffaa1c38b5","Type":"ContainerDied","Data":"33a3c2e2fc608a0159bdcfb03aa2d95414a9a30d89cb3922f1bae777eba4dbe4"} Oct 30 22:02:47.550498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.550450 5036 scope.go:122] "RemoveContainer" containerID="396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311" Oct 30 22:02:47.550607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.550584 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4" Oct 30 22:02:47.553724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.553223 5036 generic.go:356] "Generic (PLEG): container finished" podID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" containerID="5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715" exitCode=0 Oct 30 22:02:47.553724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.553327 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" Oct 30 22:02:47.553724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.553407 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" event={"ID":"52b962ac-f06e-4e57-92a5-74248dcd4eb1","Type":"ContainerDied","Data":"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715"} Oct 30 22:02:47.553724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.553449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst" event={"ID":"52b962ac-f06e-4e57-92a5-74248dcd4eb1","Type":"ContainerDied","Data":"fe013f24c245d33c7b482d749717946c39feb1c002eb5cbd4de3a4fb980f699c"} Oct 30 22:02:47.566859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.566765 5036 scope.go:122] "RemoveContainer" containerID="396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311" Oct 30 22:02:47.572403 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:47.567571 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311\": container with ID starting with 396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311 not found: ID does not exist" containerID="396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311" Oct 30 22:02:47.572403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.567607 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311"} err="failed to get container status \"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311\": rpc error: code = NotFound desc = could not find container \"396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311\": container with ID starting with 396d2f907fe9f22fa35c71d8488aac92b319d7d24af1daaaad629d50b830c311 not found: ID does not exist" Oct 30 22:02:47.572403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.567630 5036 scope.go:122] "RemoveContainer" containerID="5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715" Oct 30 22:02:47.593527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.593511 5036 scope.go:122] "RemoveContainer" containerID="5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715" Oct 30 22:02:47.594324 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:47.594249 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715\": container with ID starting with 5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715 not found: ID does not exist" containerID="5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715" Oct 30 22:02:47.594324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.594312 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715"} err="failed to get container status \"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715\": rpc error: code = NotFound desc = could not find container \"5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715\": container with ID starting with 5c6e09309a628658dc3d3dc822f08de074b12d434f88b1417ea700d7e00ec715 not found: ID does not exist" Oct 30 22:02:47.596253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.596198 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 22:02:47.601761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.601737 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3670-f9f87fb8d-zkdst"] Oct 30 22:02:47.609498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.609476 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 22:02:47.641288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:47.614945 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-6544ff699-9cvj4"] Oct 30 22:02:48.065558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.065520 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:48.065739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.065584 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:48.099550 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:48.099511 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:02:48.189281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.189235 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c655c7-9d5e-482d-8225-d2c0fdb41f8e" path="/var/lib/kubelet/pods/01c655c7-9d5e-482d-8225-d2c0fdb41f8e/volumes" Oct 30 22:02:48.189722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.189705 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" path="/var/lib/kubelet/pods/23fb2247-fb37-4e64-81df-47ffaa1c38b5/volumes" Oct 30 22:02:48.190077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.190064 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" path="/var/lib/kubelet/pods/52b962ac-f06e-4e57-92a5-74248dcd4eb1/volumes" Oct 30 22:02:48.581692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.581483 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" podUID="95b456ae-17ba-4566-9748-6c7c148879d3" containerName="payload-processing" containerID="cri-o://e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba" gracePeriod=25 Oct 30 22:02:48.587304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.587080 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" podUID="b4eadb20-bf8b-4a48-acb1-f4409370a990" containerName="payload-pre-processing" containerID="cri-o://c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764" gracePeriod=25 Oct 30 22:02:48.722410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.722327 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt"] Oct 30 22:02:48.790835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:48.790430 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a1ead7-7jlxt"] Oct 30 22:02:49.079565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.079531 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:49.079828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.079803 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:49.272581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.272519 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:02:49.382843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.382815 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:02:49.402882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.402852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\") pod \"95b456ae-17ba-4566-9748-6c7c148879d3\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " Oct 30 22:02:49.403049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.403027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\" (UniqueName: \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\") pod \"95b456ae-17ba-4566-9748-6c7c148879d3\" (UID: \"95b456ae-17ba-4566-9748-6c7c148879d3\") " Oct 30 22:02:49.404212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.404186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume" pod "95b456ae-17ba-4566-9748-6c7c148879d3" (UID: "95b456ae-17ba-4566-9748-6c7c148879d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:49.406404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.406377 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv" pod "95b456ae-17ba-4566-9748-6c7c148879d3" (UID: "95b456ae-17ba-4566-9748-6c7c148879d3"). InnerVolumeSpecName "kube-api-access-kjpzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:49.504752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.504726 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\" (UniqueName: \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\") pod \"b4eadb20-bf8b-4a48-acb1-f4409370a990\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " Oct 30 22:02:49.505476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.505458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\") pod \"b4eadb20-bf8b-4a48-acb1-f4409370a990\" (UID: \"b4eadb20-bf8b-4a48-acb1-f4409370a990\") " Oct 30 22:02:49.505792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.505770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume" pod "b4eadb20-bf8b-4a48-acb1-f4409370a990" (UID: "b4eadb20-bf8b-4a48-acb1-f4409370a990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:49.506148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.506132 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b456ae-17ba-4566-9748-6c7c148879d3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:49.506240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.506232 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpzv\" (UniqueName: \"kubernetes.io/projected/95b456ae-17ba-4566-9748-6c7c148879d3-kube-api-access-kjpzv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:49.506404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.506392 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eadb20-bf8b-4a48-acb1-f4409370a990-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:49.507014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.506989 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f" pod "b4eadb20-bf8b-4a48-acb1-f4409370a990" (UID: "b4eadb20-bf8b-4a48-acb1-f4409370a990"). InnerVolumeSpecName "kube-api-access-zjm8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:49.608170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.608087 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm8f\" (UniqueName: \"kubernetes.io/projected/b4eadb20-bf8b-4a48-acb1-f4409370a990-kube-api-access-zjm8f\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:49.707470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.707435 5036 generic.go:356] "Generic (PLEG): container finished" podID="b4eadb20-bf8b-4a48-acb1-f4409370a990" containerID="c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764" exitCode=0 Oct 30 22:02:49.707637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.707488 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" Oct 30 22:02:49.707637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.707525 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" event={"ID":"b4eadb20-bf8b-4a48-acb1-f4409370a990","Type":"ContainerDied","Data":"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764"} Oct 30 22:02:49.707637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.707582 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9" event={"ID":"b4eadb20-bf8b-4a48-acb1-f4409370a990","Type":"ContainerDied","Data":"570ea34f3d522270b31a01ccded53a5faa11d09aaf4b581bb133776bd9092541"} Oct 30 22:02:49.707637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.707613 5036 scope.go:122] "RemoveContainer" containerID="c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764" Oct 30 22:02:49.709334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.709252 5036 generic.go:356] "Generic (PLEG): container finished" podID="95b456ae-17ba-4566-9748-6c7c148879d3" containerID="e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba" exitCode=0 Oct 30 22:02:49.709489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.709336 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" Oct 30 22:02:49.709489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.709474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" event={"ID":"95b456ae-17ba-4566-9748-6c7c148879d3","Type":"ContainerDied","Data":"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba"} Oct 30 22:02:49.709626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.709513 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4" event={"ID":"95b456ae-17ba-4566-9748-6c7c148879d3","Type":"ContainerDied","Data":"d494e00d82bd32b61a8c286d71beb4ca0736c7b2f01492a720402795327300e7"} Oct 30 22:02:49.800123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.800020 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:02:49.809518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.809483 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a1ead7-b8f49d687-cqfz4"] Oct 30 22:02:49.819479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.819453 5036 scope.go:122] "RemoveContainer" containerID="c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764" Oct 30 22:02:49.820102 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:49.820076 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764\": container with ID starting with c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764 not found: ID does not exist" containerID="c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764" Oct 30 22:02:49.820293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.820246 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764"} err="failed to get container status \"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764\": rpc error: code = NotFound desc = could not find container \"c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764\": container with ID starting with c07ae1058d362589d1f538556dc062f71d616c53ab313ddc14172c66a9fe2764 not found: ID does not exist" Oct 30 22:02:49.820428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.820413 5036 scope.go:122] "RemoveContainer" containerID="e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba" Oct 30 22:02:49.854296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.849319 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:02:49.864351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.862963 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a1ead7-7c9d5fdfc5-zhzl9"] Oct 30 22:02:49.868204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.868188 5036 scope.go:122] "RemoveContainer" containerID="e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba" Oct 30 22:02:49.868595 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:49.868575 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba\": container with ID starting with e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba not found: ID does not exist" containerID="e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba" Oct 30 22:02:49.868641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:49.868604 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba"} err="failed to get container status \"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba\": rpc error: code = NotFound desc = could not find container \"e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba\": container with ID starting with e54eeb2990ab9d590f56d6c1072cab710a01ea5b2461772ebe7e60cb7c14e2ba not found: ID does not exist" Oct 30 22:02:50.065871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:50.065842 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:50.066055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:50.065894 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:50.190375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:50.190345 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9" path="/var/lib/kubelet/pods/0fbb1c13-79f0-47c4-80cd-32a1b76f6ef9/volumes" Oct 30 22:02:50.190854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:50.190838 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b456ae-17ba-4566-9748-6c7c148879d3" path="/var/lib/kubelet/pods/95b456ae-17ba-4566-9748-6c7c148879d3/volumes" Oct 30 22:02:50.191275 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:50.191252 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eadb20-bf8b-4a48-acb1-f4409370a990" path="/var/lib/kubelet/pods/b4eadb20-bf8b-4a48-acb1-f4409370a990/volumes" Oct 30 22:02:51.066302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:51.066253 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:02:51.066704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:51.066321 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 22:02:52.135217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.135177 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:02:52.136159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.136114 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" containerID="cri-o://046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8" gracePeriod=30 Oct 30 22:02:52.565339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.565015 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:52.573355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.572428 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.672349 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf"] Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673486 5036 cpu_manager.go:414] "containerMap: removing container" podUID="95b456ae-17ba-4566-9748-6c7c148879d3" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673506 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b456ae-17ba-4566-9748-6c7c148879d3" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673524 5036 cpu_manager.go:414] "containerMap: removing container" podUID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673534 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673545 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ded3a111-52af-4ca7-beea-2779769345f6" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673553 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded3a111-52af-4ca7-beea-2779769345f6" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673565 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673573 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673650 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b1e10802-7f82-4a19-992f-9668811d03dd" containerName="revoke-keys" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673659 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e10802-7f82-4a19-992f-9668811d03dd" containerName="revoke-keys" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673678 5036 cpu_manager.go:414] "containerMap: removing container" podUID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673686 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673704 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b4eadb20-bf8b-4a48-acb1-f4409370a990" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673712 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eadb20-bf8b-4a48-acb1-f4409370a990" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673868 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eadb20-bf8b-4a48-acb1-f4409370a990" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673884 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b456ae-17ba-4566-9748-6c7c148879d3" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673895 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fb2247-fb37-4e64-81df-47ffaa1c38b5" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673905 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded3a111-52af-4ca7-beea-2779769345f6" containerName="payload-pre-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673915 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc56d38-0c7c-4eb6-bcc4-9b160ffcaddb" containerName="payload-processing" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673933 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e10802-7f82-4a19-992f-9668811d03dd" containerName="revoke-keys" Oct 30 22:02:52.677142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.673943 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b962ac-f06e-4e57-92a5-74248dcd4eb1" containerName="payload-processing" Oct 30 22:02:52.697873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.690793 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:52.720448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.706920 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:52.720917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.720896 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:02:52.721296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.721255 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:52.724463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.723757 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:52.823370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.808468 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf"] Oct 30 22:02:52.864987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.864948 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:52.878422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.871380 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsml\" (UniqueName: \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:52.878422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:52.871484 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.009087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.009056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.009413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.009371 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsml\" (UniqueName: \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.021285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.021244 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.022620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.022582 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.032706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.032200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.105783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.094975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsml\" (UniqueName: \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\") pod \"maas-api-revoke-sub-0c3d6381-nqbmf\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:53.409524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:53.409489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:54.164179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:54.164141 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf"] Oct 30 22:02:54.867597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:54.866645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" event={"ID":"ac9f0f1e-2d1c-4b18-ad84-606071208d9a","Type":"ContainerStarted","Data":"e847f8e4f9deaa932db67967347809821bb1eb55ac2107775dc4f5938f2c4681"} Oct 30 22:02:54.867597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:54.866709 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" event={"ID":"ac9f0f1e-2d1c-4b18-ad84-606071208d9a","Type":"ContainerStarted","Data":"08631d5e39496e4ad48cc8e989e04245142b898132914abee329aa65418b6a4a"} Oct 30 22:02:54.895025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:54.894951 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" podStartSLOduration=2.894926924 podStartE2EDuration="2.894926924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:54.889454113 +0000 UTC m=+1398.839409082" watchObservedRunningTime="2026-10-30 22:02:54.894926924 +0000 UTC m=+1398.844881871" Oct 30 22:02:55.006820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.006789 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq"] Oct 30 22:02:55.011577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.011550 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.027687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.023547 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq"] Oct 30 22:02:55.200681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.200634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pktc\" (UniqueName: \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.200886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.200817 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.201082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.201059 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.324030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.309172 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pktc\" (UniqueName: \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.324030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.309391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.324030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.309659 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.324030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.314253 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.325448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.324882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.348125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.348060 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pktc\" (UniqueName: \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\") pod \"maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.349465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.349099 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 22:02:55.349465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.349149 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:02:55.645639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.645606 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:55.879469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.879429 5036 generic.go:356] "Generic (PLEG): container finished" podID="ac9f0f1e-2d1c-4b18-ad84-606071208d9a" containerID="e847f8e4f9deaa932db67967347809821bb1eb55ac2107775dc4f5938f2c4681" exitCode=0 Oct 30 22:02:55.879857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.879495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" event={"ID":"ac9f0f1e-2d1c-4b18-ad84-606071208d9a","Type":"ContainerDied","Data":"e847f8e4f9deaa932db67967347809821bb1eb55ac2107775dc4f5938f2c4681"} Oct 30 22:02:55.979466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:55.979321 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq"] Oct 30 22:02:55.982437 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:55.982402 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208972a1_efee_4518_8a7d_808aa3fc9226.slice/crio-87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd WatchSource:0}: Error finding container 87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd: Status 404 returned error can't find the container with id 87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd Oct 30 22:02:56.900353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:56.900314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" event={"ID":"208972a1-efee-4518-8a7d-808aa3fc9226","Type":"ContainerStarted","Data":"462800c268400baaf662a6a56f0d74f0ca53cdcf3320d2be94e9b68b6c9389bf"} Oct 30 22:02:56.900792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:56.900406 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" event={"ID":"208972a1-efee-4518-8a7d-808aa3fc9226","Type":"ContainerStarted","Data":"87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd"} Oct 30 22:02:56.924530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:56.924471 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" podStartSLOduration=2.9244557589999998 podStartE2EDuration="2.924455759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:56.921036993 +0000 UTC m=+1400.870991942" watchObservedRunningTime="2026-10-30 22:02:56.924455759 +0000 UTC m=+1400.874410737" Oct 30 22:02:57.087556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.087531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:57.263576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.262622 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\" (UniqueName: \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\") pod \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " Oct 30 22:02:57.263576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.262705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\") pod \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " Oct 30 22:02:57.263576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.262961 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\") pod \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\" (UID: \"ac9f0f1e-2d1c-4b18-ad84-606071208d9a\") " Oct 30 22:02:57.274288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.268408 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca" pod "ac9f0f1e-2d1c-4b18-ad84-606071208d9a" (UID: "ac9f0f1e-2d1c-4b18-ad84-606071208d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:57.372907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.372845 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.380321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.378715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml" pod "ac9f0f1e-2d1c-4b18-ad84-606071208d9a" (UID: "ac9f0f1e-2d1c-4b18-ad84-606071208d9a"). InnerVolumeSpecName "kube-api-access-9zsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:57.386451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.386379 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca" pod "ac9f0f1e-2d1c-4b18-ad84-606071208d9a" (UID: "ac9f0f1e-2d1c-4b18-ad84-606071208d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:57.469353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.466440 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:02:57.469353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.467642 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ac9f0f1e-2d1c-4b18-ad84-606071208d9a" containerName="revoke-keys" Oct 30 22:02:57.469353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.467665 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9f0f1e-2d1c-4b18-ad84-606071208d9a" containerName="revoke-keys" Oct 30 22:02:57.469353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.467883 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9f0f1e-2d1c-4b18-ad84-606071208d9a" containerName="revoke-keys" Oct 30 22:02:57.484395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.479134 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zsml\" (UniqueName: \"kubernetes.io/projected/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-kube-api-access-9zsml\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.484395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.479169 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9f0f1e-2d1c-4b18-ad84-606071208d9a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.503813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.498687 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:02:57.503813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.498846 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.507068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.506313 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d81a2-openshift-default-dockercfg-jlhhw\"" Oct 30 22:02:57.639648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.639616 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:02:57.639873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.639846 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" containerID="cri-o://ea0bc7207715166a9cdb6445d370dbc8bd8b1278e27c9b75e5dfc8f6e56da258" gracePeriod=30 Oct 30 22:02:57.681310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681276 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681411 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681465 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681586 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681756 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.681974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.681785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.682162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.682049 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pdp\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.704410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.704382 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:57.784346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784220 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.784346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784303 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784597 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pdp\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784643 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784680 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784719 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.788292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.784901 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.816293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.801859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.816293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.802283 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.816293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.802565 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.816293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.802626 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.816293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.802916 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.862989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.862928 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.863907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.863872 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.867714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.867659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.868458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.868411 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pdp\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:57.908994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.908964 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp"] Oct 30 22:02:57.909927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.909899 5036 cpu_manager.go:414] "containerMap: removing container" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.910050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.909928 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.910050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.909945 5036 cpu_manager.go:414] "containerMap: removing container" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.910050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.909952 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.910214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.910165 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.910214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.910180 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerName="istio-proxy" Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922364 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922412 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922443 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922653 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.923636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.922784 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\") pod \"28c5155a-5f2e-424e-bf79-4cd926c87347\" (UID: \"28c5155a-5f2e-424e-bf79-4cd926c87347\") " Oct 30 22:02:57.924304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.923730 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.927001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.926979 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp"] Oct 30 22:02:57.927361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.927343 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:57.928073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.927688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.928073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.927873 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.928073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.928035 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.933399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.933319 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:57.933826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.933775 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.934565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.934530 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:57.939130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.935371 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "kube-api-access-5jpnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:57.939130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.937497 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert" pod "28c5155a-5f2e-424e-bf79-4cd926c87347" (UID: "28c5155a-5f2e-424e-bf79-4cd926c87347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:57.941712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.941511 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" event={"ID":"ac9f0f1e-2d1c-4b18-ad84-606071208d9a","Type":"ContainerDied","Data":"08631d5e39496e4ad48cc8e989e04245142b898132914abee329aa65418b6a4a"} Oct 30 22:02:57.941712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.941567 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08631d5e39496e4ad48cc8e989e04245142b898132914abee329aa65418b6a4a" Oct 30 22:02:57.941712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.941642 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf" Oct 30 22:02:57.954188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.953715 5036 generic.go:356] "Generic (PLEG): container finished" podID="28c5155a-5f2e-424e-bf79-4cd926c87347" containerID="046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8" exitCode=0 Oct 30 22:02:57.954188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.953827 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" Oct 30 22:02:57.954188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.953974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerDied","Data":"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8"} Oct 30 22:02:57.954188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.954025 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm" event={"ID":"28c5155a-5f2e-424e-bf79-4cd926c87347","Type":"ContainerDied","Data":"46e5d04911ffd21ff41ff08d024313941ecd5652b2cd91b668c113d117b8fdae"} Oct 30 22:02:57.954188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.954053 5036 scope.go:122] "RemoveContainer" containerID="046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8" Oct 30 22:02:57.970249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.970224 5036 generic.go:356] "Generic (PLEG): container finished" podID="208972a1-efee-4518-8a7d-808aa3fc9226" containerID="462800c268400baaf662a6a56f0d74f0ca53cdcf3320d2be94e9b68b6c9389bf" exitCode=0 Oct 30 22:02:57.970474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.970457 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" event={"ID":"208972a1-efee-4518-8a7d-808aa3fc9226","Type":"ContainerDied","Data":"462800c268400baaf662a6a56f0d74f0ca53cdcf3320d2be94e9b68b6c9389bf"} Oct 30 22:02:57.986543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:57.984411 5036 scope.go:122] "RemoveContainer" containerID="d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082" Oct 30 22:02:58.013478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.013452 5036 scope.go:122] "RemoveContainer" containerID="046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8" Oct 30 22:02:58.013805 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:58.013779 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8\": container with ID starting with 046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8 not found: ID does not exist" containerID="046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8" Oct 30 22:02:58.013886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.013818 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8"} err="failed to get container status \"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8\": rpc error: code = NotFound desc = could not find container \"046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8\": container with ID starting with 046b5dd0e9b60a3d5bd277319cfc227410fa5f3ac04056046de7f4f00295a5b8 not found: ID does not exist" Oct 30 22:02:58.013886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.013846 5036 scope.go:122] "RemoveContainer" containerID="d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082" Oct 30 22:02:58.014238 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:02:58.014211 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082\": container with ID starting with d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082 not found: ID does not exist" containerID="d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082" Oct 30 22:02:58.014377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.014248 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082"} err="failed to get container status \"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082\": rpc error: code = NotFound desc = could not find container \"d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082\": container with ID starting with d08fc9d637080f5d539c8808ffaa82b89617dc811895a4fcff87279f666dd082 not found: ID does not exist" Oct 30 22:02:58.025668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.025668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025614 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkcq\" (UniqueName: \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025759 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025770 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025782 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c5155a-5f2e-424e-bf79-4cd926c87347-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025791 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpnr\" (UniqueName: \"kubernetes.io/projected/28c5155a-5f2e-424e-bf79-4cd926c87347-kube-api-access-5jpnr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025800 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025808 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025817 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025826 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.025987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.025834 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c5155a-5f2e-424e-bf79-4cd926c87347-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.031356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.030249 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:02:58.034339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.034281 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c097ae-openshift-default-746f759777-rppsm"] Oct 30 22:02:58.127113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.127020 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.127113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.127079 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkcq\" (UniqueName: \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.127411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.127174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.127480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.127447 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:58.127905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.127881 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.129700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.129682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.139370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.139337 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkcq\" (UniqueName: \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\") pod \"maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.193587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.193551 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c5155a-5f2e-424e-bf79-4cd926c87347" path="/var/lib/kubelet/pods/28c5155a-5f2e-424e-bf79-4cd926c87347/volumes" Oct 30 22:02:58.253868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.253831 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:02:58.422233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.422200 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:02:58.426812 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:02:58.426783 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd355ee_d65d_44b6_9577_514a80a87e1d.slice/crio-46fbeb60c0a381d6fd1153d157f3e6997a449491f9a238af84bbe165b2c46abc WatchSource:0}: Error finding container 46fbeb60c0a381d6fd1153d157f3e6997a449491f9a238af84bbe165b2c46abc: Status 404 returned error can't find the container with id 46fbeb60c0a381d6fd1153d157f3e6997a449491f9a238af84bbe165b2c46abc Oct 30 22:02:58.430593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.430556 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:58.430686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.430644 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:58.430729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.430687 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:58.548883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.548853 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp"] Oct 30 22:02:58.984331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.984293 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerStarted","Data":"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202"} Oct 30 22:02:58.984817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.984392 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerStarted","Data":"46fbeb60c0a381d6fd1153d157f3e6997a449491f9a238af84bbe165b2c46abc"} Oct 30 22:02:58.985767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:58.985745 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" event={"ID":"a9b12658-620d-42df-8ecf-2d35bb50858e","Type":"ContainerStarted","Data":"2078dca558223e9a0c793b313ce8c422404b0acfb90c526b5b30a6a56f89a77f"} Oct 30 22:02:59.018331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.018251 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podStartSLOduration=2.018232031 podStartE2EDuration="2.018232031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:59.011707774 +0000 UTC m=+1402.961662747" watchObservedRunningTime="2026-10-30 22:02:59.018232031 +0000 UTC m=+1402.968186977" Oct 30 22:02:59.038544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.038515 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6"] Oct 30 22:02:59.044846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.044815 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f26c672-ssfq6"] Oct 30 22:02:59.130843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.130669 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:59.177287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.175793 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:59.230439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.230412 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:59.347681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.347647 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\" (UniqueName: \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\") pod \"208972a1-efee-4518-8a7d-808aa3fc9226\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " Oct 30 22:02:59.347887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.347873 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\") pod \"208972a1-efee-4518-8a7d-808aa3fc9226\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " Oct 30 22:02:59.347933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.347915 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\") pod \"208972a1-efee-4518-8a7d-808aa3fc9226\" (UID: \"208972a1-efee-4518-8a7d-808aa3fc9226\") " Oct 30 22:02:59.348319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.348245 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca" pod "208972a1-efee-4518-8a7d-808aa3fc9226" (UID: "208972a1-efee-4518-8a7d-808aa3fc9226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:59.348801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.348778 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:59.350206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.350165 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc" pod "208972a1-efee-4518-8a7d-808aa3fc9226" (UID: "208972a1-efee-4518-8a7d-808aa3fc9226"). InnerVolumeSpecName "kube-api-access-7pktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:59.350692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.350671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca" pod "208972a1-efee-4518-8a7d-808aa3fc9226" (UID: "208972a1-efee-4518-8a7d-808aa3fc9226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:59.456477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.453094 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208972a1-efee-4518-8a7d-808aa3fc9226-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:59.456477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.453140 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pktc\" (UniqueName: \"kubernetes.io/projected/208972a1-efee-4518-8a7d-808aa3fc9226-kube-api-access-7pktc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:59.996107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.996070 5036 generic.go:356] "Generic (PLEG): container finished" podID="a9b12658-620d-42df-8ecf-2d35bb50858e" containerID="a254cf34dd8779f8723d162245e955c9c8921e89b9a63bc1a1ab1c8f97b51600" exitCode=0 Oct 30 22:02:59.996593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.996159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" event={"ID":"a9b12658-620d-42df-8ecf-2d35bb50858e","Type":"ContainerDied","Data":"a254cf34dd8779f8723d162245e955c9c8921e89b9a63bc1a1ab1c8f97b51600"} Oct 30 22:02:59.997910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.997891 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" Oct 30 22:02:59.998034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.997938 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq" event={"ID":"208972a1-efee-4518-8a7d-808aa3fc9226","Type":"ContainerDied","Data":"87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd"} Oct 30 22:02:59.998034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.997978 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e601b42087af1d0cdcb91333b8905af28e56ca99335550b127dd7d859cd7cd" Oct 30 22:02:59.998320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.998304 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:02:59.999323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:02:59.999303 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:03:00.189958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.189922 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec60992-5bf6-426a-b362-f627124db07c" path="/var/lib/kubelet/pods/1ec60992-5bf6-426a-b362-f627124db07c/volumes" Oct 30 22:03:00.539947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.539913 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw"] Oct 30 22:03:00.540699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.540676 5036 cpu_manager.go:414] "containerMap: removing container" podUID="208972a1-efee-4518-8a7d-808aa3fc9226" containerName="revoke-keys" Oct 30 22:03:00.540699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.540699 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208972a1-efee-4518-8a7d-808aa3fc9226" containerName="revoke-keys" Oct 30 22:03:00.540894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.540881 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="208972a1-efee-4518-8a7d-808aa3fc9226" containerName="revoke-keys" Oct 30 22:03:00.544856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.544833 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.558602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.557081 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw"] Oct 30 22:03:00.676751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.676721 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf8q\" (UniqueName: \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.676973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.676870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.677034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.677003 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.778953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.778915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf8q\" (UniqueName: \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.779154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.779010 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.779154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.779076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.779803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.779746 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.781803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.781776 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.790934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.790873 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf8q\" (UniqueName: \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\") pod \"maas-api-revoke-sub-873a077c-vn5vw\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:00.864451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:00.864419 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:01.161088 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:01.161059 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c2b547_cf90_4e2f_9e6c_ad90ed993d8b.slice/crio-ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215 WatchSource:0}: Error finding container ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215: Status 404 returned error can't find the container with id ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215 Oct 30 22:03:01.163150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.163038 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw"] Oct 30 22:03:01.178259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.178233 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:03:01.289899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.288819 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\" (UniqueName: \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\") pod \"a9b12658-620d-42df-8ecf-2d35bb50858e\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " Oct 30 22:03:01.290712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.290686 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\") pod \"a9b12658-620d-42df-8ecf-2d35bb50858e\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " Oct 30 22:03:01.291307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.291288 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\") pod \"a9b12658-620d-42df-8ecf-2d35bb50858e\" (UID: \"a9b12658-620d-42df-8ecf-2d35bb50858e\") " Oct 30 22:03:01.294885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.291147 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca" pod "a9b12658-620d-42df-8ecf-2d35bb50858e" (UID: "a9b12658-620d-42df-8ecf-2d35bb50858e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:01.297167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.296887 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca" pod "a9b12658-620d-42df-8ecf-2d35bb50858e" (UID: "a9b12658-620d-42df-8ecf-2d35bb50858e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:01.298094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.298071 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq" pod "a9b12658-620d-42df-8ecf-2d35bb50858e" (UID: "a9b12658-620d-42df-8ecf-2d35bb50858e"). InnerVolumeSpecName "kube-api-access-pjkcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:01.394310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.394258 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:01.394310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.394306 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b12658-620d-42df-8ecf-2d35bb50858e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:01.394519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:01.394318 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkcq\" (UniqueName: \"kubernetes.io/projected/a9b12658-620d-42df-8ecf-2d35bb50858e-kube-api-access-pjkcq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:02.020166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.020132 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" event={"ID":"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b","Type":"ContainerStarted","Data":"af92070dd2eeb9c126ba92e85e85fdce0228e1c912c4aadbfc7017a96ac77f84"} Oct 30 22:03:02.020496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.020472 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" event={"ID":"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b","Type":"ContainerStarted","Data":"ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215"} Oct 30 22:03:02.022210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.022187 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" Oct 30 22:03:02.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.022238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp" event={"ID":"a9b12658-620d-42df-8ecf-2d35bb50858e","Type":"ContainerDied","Data":"2078dca558223e9a0c793b313ce8c422404b0acfb90c526b5b30a6a56f89a77f"} Oct 30 22:03:02.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.022308 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2078dca558223e9a0c793b313ce8c422404b0acfb90c526b5b30a6a56f89a77f" Oct 30 22:03:02.041876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:02.038001 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" podStartSLOduration=2.037982981 podStartE2EDuration="2.037982981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:02.037829532 +0000 UTC m=+1405.987784479" watchObservedRunningTime="2026-10-30 22:03:02.037982981 +0000 UTC m=+1405.987937931" Oct 30 22:03:03.123613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.117477 5036 generic.go:356] "Generic (PLEG): container finished" podID="72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" containerID="af92070dd2eeb9c126ba92e85e85fdce0228e1c912c4aadbfc7017a96ac77f84" exitCode=0 Oct 30 22:03:03.123613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.117635 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" event={"ID":"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b","Type":"ContainerDied","Data":"af92070dd2eeb9c126ba92e85e85fdce0228e1c912c4aadbfc7017a96ac77f84"} Oct 30 22:03:03.137784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.137744 5036 generic.go:356] "Generic (PLEG): container finished" podID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerID="ea0bc7207715166a9cdb6445d370dbc8bd8b1278e27c9b75e5dfc8f6e56da258" exitCode=0 Oct 30 22:03:03.138050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.138035 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerDied","Data":"ea0bc7207715166a9cdb6445d370dbc8bd8b1278e27c9b75e5dfc8f6e56da258"} Oct 30 22:03:03.138159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.138149 5036 scope.go:122] "RemoveContainer" containerID="f70b851585945a87dbe6f4e661b13826a02c64c7ced1f255dfac15ee6acc925a" Oct 30 22:03:03.179676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.175910 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:03:03.191996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.191963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192037 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192303 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:03.192372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192318 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192385 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192478 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:03.192601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192631 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\") pod \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\" (UID: \"dddb6e14-ec47-48e7-b3c2-0fd34846fef0\") " Oct 30 22:03:03.192988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.192739 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:03.193048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193035 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:03.193337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193317 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.193337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193336 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.193525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193345 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.193525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193354 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.193783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.193608 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:03.195465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.195436 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:03.195700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.195594 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "kube-api-access-tn4fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:03.196024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.196000 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:03.196710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.196687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token" pod "dddb6e14-ec47-48e7-b3c2-0fd34846fef0" (UID: "dddb6e14-ec47-48e7-b3c2-0fd34846fef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:03.294764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.294731 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.294764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.294758 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.294764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.294767 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4fs\" (UniqueName: \"kubernetes.io/projected/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-kube-api-access-tn4fs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.295007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.294778 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.295007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:03.294787 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dddb6e14-ec47-48e7-b3c2-0fd34846fef0-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.151594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.151549 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" event={"ID":"dddb6e14-ec47-48e7-b3c2-0fd34846fef0","Type":"ContainerDied","Data":"a9a0db5d15e1957cbb8fa241d9f5a8d7671c14a9146a195db390672a94348d40"} Oct 30 22:03:04.152071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.151640 5036 scope.go:122] "RemoveContainer" containerID="ea0bc7207715166a9cdb6445d370dbc8bd8b1278e27c9b75e5dfc8f6e56da258" Oct 30 22:03:04.152071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.151833 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c" Oct 30 22:03:04.192833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.192810 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:03:04.201774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.201187 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a1ead7-openshift-default-568b767f7-hqq8c"] Oct 30 22:03:04.403232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.403166 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:04.453502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.453468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\") pod \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " Oct 30 22:03:04.453672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.453520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\" (UniqueName: \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\") pod \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " Oct 30 22:03:04.453672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.453546 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\") pod \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\" (UID: \"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b\") " Oct 30 22:03:04.454342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.454314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca" pod "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" (UID: "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.456554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.456525 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q" pod "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" (UID: "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b"). InnerVolumeSpecName "kube-api-access-qcf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:04.472987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.472952 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca" pod "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" (UID: "72c2b547-cf90-4e2f-9e6c-ad90ed993d8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.555865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.555828 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.555865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.555866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcf8q\" (UniqueName: \"kubernetes.io/projected/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-kube-api-access-qcf8q\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.556074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:04.555879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:05.124191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124160 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w"] Oct 30 22:03:05.124857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124841 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.124919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124859 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.124919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124894 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.124919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124902 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.125019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124920 5036 cpu_manager.go:414] "containerMap: removing container" podUID="72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" containerName="revoke-keys" Oct 30 22:03:05.125019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124925 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" containerName="revoke-keys" Oct 30 22:03:05.125019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124934 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a9b12658-620d-42df-8ecf-2d35bb50858e" containerName="revoke-keys" Oct 30 22:03:05.125019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.124938 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b12658-620d-42df-8ecf-2d35bb50858e" containerName="revoke-keys" Oct 30 22:03:05.125188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.125079 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" containerName="revoke-keys" Oct 30 22:03:05.125188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.125090 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b12658-620d-42df-8ecf-2d35bb50858e" containerName="revoke-keys" Oct 30 22:03:05.125188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.125104 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.125188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.125110 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" containerName="istio-proxy" Oct 30 22:03:05.129012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.128994 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.144965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.144940 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w"] Oct 30 22:03:05.166863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.166825 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvns\" (UniqueName: \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.167320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.166913 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.167320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.167004 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.171689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.171662 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" Oct 30 22:03:05.171823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.171708 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw" event={"ID":"72c2b547-cf90-4e2f-9e6c-ad90ed993d8b","Type":"ContainerDied","Data":"ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215"} Oct 30 22:03:05.171823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.171782 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec105b4909d8927f49f6b7135b70d4cd26e4c53b807c79693606f99d80763215" Oct 30 22:03:05.273117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.269156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.273117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.269259 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.273117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.269541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvns\" (UniqueName: \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.273117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.270794 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.284610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.283760 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.296288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.295664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvns\" (UniqueName: \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\") pod \"maas-api-revoke-sub-773950ef-8mp9w\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.444757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.444726 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:05.983306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:05.983259 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w"] Oct 30 22:03:05.984168 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:05.984139 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f65470_ee9b_42c5_a746_53c08ba3aed1.slice/crio-eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b WatchSource:0}: Error finding container eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b: Status 404 returned error can't find the container with id eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b Oct 30 22:03:06.202125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:06.202087 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddb6e14-ec47-48e7-b3c2-0fd34846fef0" path="/var/lib/kubelet/pods/dddb6e14-ec47-48e7-b3c2-0fd34846fef0/volumes" Oct 30 22:03:06.202644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:06.202582 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" event={"ID":"32f65470-ee9b-42c5-a746-53c08ba3aed1","Type":"ContainerStarted","Data":"eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b"} Oct 30 22:03:07.215186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:07.215088 5036 generic.go:356] "Generic (PLEG): container finished" podID="32f65470-ee9b-42c5-a746-53c08ba3aed1" containerID="3a5e151edeb6a1a86c2f3a151982618e2e7af73e5462a379588567d0ed159fed" exitCode=0 Oct 30 22:03:07.215186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:07.215179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" event={"ID":"32f65470-ee9b-42c5-a746-53c08ba3aed1","Type":"ContainerDied","Data":"3a5e151edeb6a1a86c2f3a151982618e2e7af73e5462a379588567d0ed159fed"} Oct 30 22:03:08.629372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.629341 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:08.775026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.774998 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\" (UniqueName: \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\") pod \"32f65470-ee9b-42c5-a746-53c08ba3aed1\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " Oct 30 22:03:08.775196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.775091 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\") pod \"32f65470-ee9b-42c5-a746-53c08ba3aed1\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " Oct 30 22:03:08.775251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.775228 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\") pod \"32f65470-ee9b-42c5-a746-53c08ba3aed1\" (UID: \"32f65470-ee9b-42c5-a746-53c08ba3aed1\") " Oct 30 22:03:08.775562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.775534 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca" pod "32f65470-ee9b-42c5-a746-53c08ba3aed1" (UID: "32f65470-ee9b-42c5-a746-53c08ba3aed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:08.775823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.775806 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:08.777193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.777169 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns" pod "32f65470-ee9b-42c5-a746-53c08ba3aed1" (UID: "32f65470-ee9b-42c5-a746-53c08ba3aed1"). InnerVolumeSpecName "kube-api-access-ftvns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:08.777252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.777205 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca" pod "32f65470-ee9b-42c5-a746-53c08ba3aed1" (UID: "32f65470-ee9b-42c5-a746-53c08ba3aed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:08.877057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.876962 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvns\" (UniqueName: \"kubernetes.io/projected/32f65470-ee9b-42c5-a746-53c08ba3aed1-kube-api-access-ftvns\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:08.877057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:08.877000 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f65470-ee9b-42c5-a746-53c08ba3aed1-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.237028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:09.236999 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" Oct 30 22:03:09.237204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:09.237038 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w" event={"ID":"32f65470-ee9b-42c5-a746-53c08ba3aed1","Type":"ContainerDied","Data":"eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b"} Oct 30 22:03:09.237204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:09.237095 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb873ed56f69953f275283ff8cc6e070b9d96f37d68827c1e02b3cf743bc287b" Oct 30 22:03:10.175282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.175228 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq"] Oct 30 22:03:10.188590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.188567 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4cae46-65fcq"] Oct 30 22:03:10.938965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.938920 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt"] Oct 30 22:03:10.939604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.939588 5036 cpu_manager.go:414] "containerMap: removing container" podUID="32f65470-ee9b-42c5-a746-53c08ba3aed1" containerName="revoke-keys" Oct 30 22:03:10.939604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.939604 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f65470-ee9b-42c5-a746-53c08ba3aed1" containerName="revoke-keys" Oct 30 22:03:10.939761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.939718 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f65470-ee9b-42c5-a746-53c08ba3aed1" containerName="revoke-keys" Oct 30 22:03:10.943518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.943497 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:10.955119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.955094 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:10.955119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.955092 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:10.955408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.955352 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:10.955648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.955628 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt"] Oct 30 22:03:10.955810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:10.955795 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:11.112181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.112142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.112462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.112369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk5j\" (UniqueName: \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.112462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.112407 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.213565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.213463 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk5j\" (UniqueName: \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.213565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.213505 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.214129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.213715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.214407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.214386 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.216393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.216357 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.228421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.228380 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk5j\" (UniqueName: \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\") pod \"maas-api-revoke-sub-62af3079-mndnt\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.267772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.267742 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:11.557466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:11.557433 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt"] Oct 30 22:03:11.558294 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:11.558248 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5617cee_15cb_4ca7_8c8b_e794bc72f5d7.slice/crio-82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682 WatchSource:0}: Error finding container 82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682: Status 404 returned error can't find the container with id 82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682 Oct 30 22:03:12.189358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:12.189329 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208972a1-efee-4518-8a7d-808aa3fc9226" path="/var/lib/kubelet/pods/208972a1-efee-4518-8a7d-808aa3fc9226/volumes" Oct 30 22:03:12.276181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:12.276142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" event={"ID":"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7","Type":"ContainerStarted","Data":"c4fe98d4fb81b9a64309f6a9c00d5d52537804d004881e8801f9d4614c94fa4b"} Oct 30 22:03:12.276776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:12.276218 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" event={"ID":"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7","Type":"ContainerStarted","Data":"82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682"} Oct 30 22:03:12.294798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:12.294733 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" podStartSLOduration=2.294712423 podStartE2EDuration="2.294712423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:12.293646862 +0000 UTC m=+1416.243601807" watchObservedRunningTime="2026-10-30 22:03:12.294712423 +0000 UTC m=+1416.244667371" Oct 30 22:03:13.083250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:13.083217 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp"] Oct 30 22:03:13.087907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:13.087878 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bca8c9-x2qtp"] Oct 30 22:03:13.361354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:13.359363 5036 generic.go:356] "Generic (PLEG): container finished" podID="f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" containerID="c4fe98d4fb81b9a64309f6a9c00d5d52537804d004881e8801f9d4614c94fa4b" exitCode=0 Oct 30 22:03:13.361354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:13.359583 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" event={"ID":"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7","Type":"ContainerDied","Data":"c4fe98d4fb81b9a64309f6a9c00d5d52537804d004881e8801f9d4614c94fa4b"} Oct 30 22:03:14.254587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:14.254385 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b12658-620d-42df-8ecf-2d35bb50858e" path="/var/lib/kubelet/pods/a9b12658-620d-42df-8ecf-2d35bb50858e/volumes" Oct 30 22:03:15.329473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.302703 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:15.461461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.385383 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\" (UniqueName: \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\") pod \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " Oct 30 22:03:15.461461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.385540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\") pod \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " Oct 30 22:03:15.461461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.385605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\") pod \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\" (UID: \"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7\") " Oct 30 22:03:15.461461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.405704 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca" pod "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" (UID: "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:15.486884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.485737 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:15.727769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.727676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" event={"ID":"f5617cee-15cb-4ca7-8c8b-e794bc72f5d7","Type":"ContainerDied","Data":"82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682"} Oct 30 22:03:15.728019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.727999 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ba4725e193d3ddaaa94fff4dc8fb3a4315ca16792889f89075107b8ab94682" Oct 30 22:03:15.728213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.728197 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt" Oct 30 22:03:15.764682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.764628 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j" pod "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" (UID: "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7"). InnerVolumeSpecName "kube-api-access-mxk5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:15.775628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.775578 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca" pod "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" (UID: "f5617cee-15cb-4ca7-8c8b-e794bc72f5d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:15.825505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.825468 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:15.826054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:15.826023 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxk5j\" (UniqueName: \"kubernetes.io/projected/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7-kube-api-access-mxk5j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:16.380649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.380549 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.411350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.401587 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.412347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.411911 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.413580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.411948 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:03:16.625501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.625453 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.698403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.698363 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:16.698693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.698660 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.781492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.781452 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.911891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:16.909771 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.125644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:17.106292 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.245298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:17.243553 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:17.245298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:17.243611 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.107886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.107843 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:03:18.109244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.109005 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" containerID="cri-o://c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0" gracePeriod=30 Oct 30 22:03:18.128971 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:03:18.128933 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:03:18.381007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.380914 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9"] Oct 30 22:03:18.381885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.381861 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" containerName="revoke-keys" Oct 30 22:03:18.382004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.381886 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" containerName="revoke-keys" Oct 30 22:03:18.382081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.382032 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" containerName="revoke-keys" Oct 30 22:03:18.386110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.386089 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.396752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.396725 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:18.397314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.397293 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:18.397433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.397355 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:18.406069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.406038 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9"] Oct 30 22:03:18.496467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.496434 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:18.543950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.543916 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczvl\" (UniqueName: \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.544152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.544050 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.544349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.544325 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.645402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.645366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczvl\" (UniqueName: \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.645596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.645470 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.645740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.645721 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.646191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.646171 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.648207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.648186 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.659112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.659085 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczvl\" (UniqueName: \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\") pod \"maas-api-revoke-sub-16dde3ef-8j6g9\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.664670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.664648 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr"] Oct 30 22:03:18.668985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.668963 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.682473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.682449 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr"] Oct 30 22:03:18.704076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.704048 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:18.848163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.848124 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.848425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.848305 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.848425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.848354 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbds6\" (UniqueName: \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.950423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.950335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.950423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.950404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.950640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.950431 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbds6\" (UniqueName: \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.951156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.951128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.952889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.952868 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.964373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.964342 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbds6\" (UniqueName: \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\") pod \"maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.993330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.993302 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:18.996832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:18.996808 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9"] Oct 30 22:03:18.997473 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:18.997445 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f87aff_9158_4c40_80d4_fe180ea44e6a.slice/crio-4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376 WatchSource:0}: Error finding container 4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376: Status 404 returned error can't find the container with id 4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376 Oct 30 22:03:19.293967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.293931 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr"] Oct 30 22:03:19.294379 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:19.294339 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2cf94b_ae2a_4617_98d6_50bc23ad2851.slice/crio-fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92 WatchSource:0}: Error finding container fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92: Status 404 returned error can't find the container with id fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92 Oct 30 22:03:19.816334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.816229 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" event={"ID":"53f87aff-9158-4c40-80d4-fe180ea44e6a","Type":"ContainerStarted","Data":"d0c718a471991bb4fc289fe21923884195ac61dc4baaa22ba13c91137914ea78"} Oct 30 22:03:19.816517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.816337 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" event={"ID":"53f87aff-9158-4c40-80d4-fe180ea44e6a","Type":"ContainerStarted","Data":"4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376"} Oct 30 22:03:19.818394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.818365 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" event={"ID":"da2cf94b-ae2a-4617-98d6-50bc23ad2851","Type":"ContainerStarted","Data":"090d7c81f42aa572a0c0219441b85b20c84bff8cff7fd65709a3c8111f0c5bf4"} Oct 30 22:03:19.818532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.818421 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" event={"ID":"da2cf94b-ae2a-4617-98d6-50bc23ad2851","Type":"ContainerStarted","Data":"fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92"} Oct 30 22:03:19.838446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.838388 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" podStartSLOduration=1.838370457 podStartE2EDuration="1.838370457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:19.832808158 +0000 UTC m=+1423.782763105" watchObservedRunningTime="2026-10-30 22:03:19.838370457 +0000 UTC m=+1423.788325405" Oct 30 22:03:19.854258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:19.854216 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" podStartSLOduration=1.854203711 podStartE2EDuration="1.854203711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:19.848860037 +0000 UTC m=+1423.798814984" watchObservedRunningTime="2026-10-30 22:03:19.854203711 +0000 UTC m=+1423.804158656" Oct 30 22:03:20.832035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:20.831932 5036 generic.go:356] "Generic (PLEG): container finished" podID="53f87aff-9158-4c40-80d4-fe180ea44e6a" containerID="d0c718a471991bb4fc289fe21923884195ac61dc4baaa22ba13c91137914ea78" exitCode=0 Oct 30 22:03:20.832568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:20.832023 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" event={"ID":"53f87aff-9158-4c40-80d4-fe180ea44e6a","Type":"ContainerDied","Data":"d0c718a471991bb4fc289fe21923884195ac61dc4baaa22ba13c91137914ea78"} Oct 30 22:03:20.834345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:20.834316 5036 generic.go:356] "Generic (PLEG): container finished" podID="da2cf94b-ae2a-4617-98d6-50bc23ad2851" containerID="090d7c81f42aa572a0c0219441b85b20c84bff8cff7fd65709a3c8111f0c5bf4" exitCode=0 Oct 30 22:03:20.834462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:20.834401 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" event={"ID":"da2cf94b-ae2a-4617-98d6-50bc23ad2851","Type":"ContainerDied","Data":"090d7c81f42aa572a0c0219441b85b20c84bff8cff7fd65709a3c8111f0c5bf4"} Oct 30 22:03:22.078360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.077632 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:22.122242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.121305 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\" (UniqueName: \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\") pod \"53f87aff-9158-4c40-80d4-fe180ea44e6a\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " Oct 30 22:03:22.122242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.121413 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\") pod \"53f87aff-9158-4c40-80d4-fe180ea44e6a\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " Oct 30 22:03:22.122242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.121440 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\") pod \"53f87aff-9158-4c40-80d4-fe180ea44e6a\" (UID: \"53f87aff-9158-4c40-80d4-fe180ea44e6a\") " Oct 30 22:03:22.122678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.122562 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca" pod "53f87aff-9158-4c40-80d4-fe180ea44e6a" (UID: "53f87aff-9158-4c40-80d4-fe180ea44e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:22.129741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.129695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl" pod "53f87aff-9158-4c40-80d4-fe180ea44e6a" (UID: "53f87aff-9158-4c40-80d4-fe180ea44e6a"). InnerVolumeSpecName "kube-api-access-fczvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:22.131403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.131371 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca" pod "53f87aff-9158-4c40-80d4-fe180ea44e6a" (UID: "53f87aff-9158-4c40-80d4-fe180ea44e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:22.159749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.159727 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:22.223179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.223143 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\") pod \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " Oct 30 22:03:22.223350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.223252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\" (UniqueName: \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\") pod \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " Oct 30 22:03:22.223350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.223345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\") pod \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\" (UID: \"da2cf94b-ae2a-4617-98d6-50bc23ad2851\") " Oct 30 22:03:22.223809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.223784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca" pod "da2cf94b-ae2a-4617-98d6-50bc23ad2851" (UID: "da2cf94b-ae2a-4617-98d6-50bc23ad2851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:22.224552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.224476 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczvl\" (UniqueName: \"kubernetes.io/projected/53f87aff-9158-4c40-80d4-fe180ea44e6a-kube-api-access-fczvl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.224552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.224501 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.224552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.224516 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.224552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.224531 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f87aff-9158-4c40-80d4-fe180ea44e6a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.226010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.225985 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca" pod "da2cf94b-ae2a-4617-98d6-50bc23ad2851" (UID: "da2cf94b-ae2a-4617-98d6-50bc23ad2851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:22.226754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.226729 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6" pod "da2cf94b-ae2a-4617-98d6-50bc23ad2851" (UID: "da2cf94b-ae2a-4617-98d6-50bc23ad2851"). InnerVolumeSpecName "kube-api-access-xbds6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:22.326093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.326055 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2cf94b-ae2a-4617-98d6-50bc23ad2851-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.326093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.326094 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbds6\" (UniqueName: \"kubernetes.io/projected/da2cf94b-ae2a-4617-98d6-50bc23ad2851-kube-api-access-xbds6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:22.568176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.568139 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:22.568378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.568210 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:22.853981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.853892 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" Oct 30 22:03:22.853981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.853921 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr" event={"ID":"da2cf94b-ae2a-4617-98d6-50bc23ad2851","Type":"ContainerDied","Data":"fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92"} Oct 30 22:03:22.853981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.853979 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc59bf7061df84d186a02fd98824c882874a2a3852c123e1e9e9017cdc177b92" Oct 30 22:03:22.856033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.856004 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" Oct 30 22:03:22.856207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.855997 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9" event={"ID":"53f87aff-9158-4c40-80d4-fe180ea44e6a","Type":"ContainerDied","Data":"4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376"} Oct 30 22:03:22.856207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:22.856144 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4566bb76048f66460258fb23dd9108eaadda926b3dae278e767fb659d1679376" Oct 30 22:03:23.658111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.658086 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:03:23.761482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.759668 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.761979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.761956 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763000 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.762905 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763061 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763088 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763150 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763306 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763429 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\") pod \"7cc042a5-0e5e-455f-8622-cf662acc6ead\" (UID: \"7cc042a5-0e5e-455f-8622-cf662acc6ead\") " Oct 30 22:03:23.764191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.763555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:23.764783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.764601 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.764783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.764623 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.764887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.764821 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:23.765841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.765787 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:23.766250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.766228 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:23.768575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.768539 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:23.772325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.772301 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:23.772654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.772621 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "kube-api-access-z9v5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:23.776096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.776070 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy" pod "7cc042a5-0e5e-455f-8622-cf662acc6ead" (UID: "7cc042a5-0e5e-455f-8622-cf662acc6ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:23.865645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865608 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865642 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865657 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865670 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc042a5-0e5e-455f-8622-cf662acc6ead-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865684 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9v5b\" (UniqueName: \"kubernetes.io/projected/7cc042a5-0e5e-455f-8622-cf662acc6ead-kube-api-access-z9v5b\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865696 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc042a5-0e5e-455f-8622-cf662acc6ead-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.865903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.865707 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc042a5-0e5e-455f-8622-cf662acc6ead-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:23.866917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.866888 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerID="c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0" exitCode=0 Oct 30 22:03:23.867023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.866941 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" Oct 30 22:03:23.867023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.866971 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerDied","Data":"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0"} Oct 30 22:03:23.867103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.867037 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml" event={"ID":"7cc042a5-0e5e-455f-8622-cf662acc6ead","Type":"ContainerDied","Data":"bbda0ce2b5957d2dbc168cbfdc177e11e65b42c628b37d229ebe06b1ac12719d"} Oct 30 22:03:23.867103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.867066 5036 scope.go:122] "RemoveContainer" containerID="c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0" Oct 30 22:03:23.880315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.880252 5036 scope.go:122] "RemoveContainer" containerID="6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737" Oct 30 22:03:23.891962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.891941 5036 scope.go:122] "RemoveContainer" containerID="c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0" Oct 30 22:03:23.892352 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:03:23.892322 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0\": container with ID starting with c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0 not found: ID does not exist" containerID="c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0" Oct 30 22:03:23.892456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.892362 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0"} err="failed to get container status \"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0\": rpc error: code = NotFound desc = could not find container \"c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0\": container with ID starting with c5d979fcfffdb9a1d60f2c817363503efd737214ea3c0d9f3063a16b8e5c76e0 not found: ID does not exist" Oct 30 22:03:23.892456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.892388 5036 scope.go:122] "RemoveContainer" containerID="6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737" Oct 30 22:03:23.892701 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:03:23.892680 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737\": container with ID starting with 6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737 not found: ID does not exist" containerID="6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737" Oct 30 22:03:23.892764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.892711 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737"} err="failed to get container status \"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737\": rpc error: code = NotFound desc = could not find container \"6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737\": container with ID starting with 6a67982a1a10a4b14d25bcda304bdb574f23ef815f2d15bcc49046c72c404737 not found: ID does not exist" Oct 30 22:03:23.907077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.904285 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:03:23.920714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.920684 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:03:23.921830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921800 5036 cpu_manager.go:414] "containerMap: removing container" podUID="da2cf94b-ae2a-4617-98d6-50bc23ad2851" containerName="revoke-keys" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921832 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2cf94b-ae2a-4617-98d6-50bc23ad2851" containerName="revoke-keys" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921850 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921857 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921884 5036 cpu_manager.go:414] "containerMap: removing container" podUID="53f87aff-9158-4c40-80d4-fe180ea44e6a" containerName="revoke-keys" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921892 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f87aff-9158-4c40-80d4-fe180ea44e6a" containerName="revoke-keys" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921909 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.921932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.921916 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.922304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.922068 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2cf94b-ae2a-4617-98d6-50bc23ad2851" containerName="revoke-keys" Oct 30 22:03:23.922304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.922081 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.922304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.922094 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f87aff-9158-4c40-80d4-fe180ea44e6a" containerName="revoke-keys" Oct 30 22:03:23.922304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.922103 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" containerName="istio-proxy" Oct 30 22:03:23.927484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.927461 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cae46-openshift-default-65d7cd8d9-b6tml"] Oct 30 22:03:23.927640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.927621 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:23.938309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.936354 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:03:23.983066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.983031 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4"] Oct 30 22:03:23.996210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:23.996188 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.002847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.002821 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:24.005258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.005236 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:24.007311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.007288 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4"] Oct 30 22:03:24.042118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.042050 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:24.042329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.042095 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:24.068036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068004 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068044 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxppk\" (UniqueName: \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.068221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068177 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068230 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068414 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068439 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.068660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068495 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.068660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068608 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068644 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.068815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.068744 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgn9h\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.169819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.169779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.169819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.169822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxppk\" (UniqueName: \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.170062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.169857 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.169900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.169945 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170044 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.170314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.170314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170257 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170295 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170408 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgn9h\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.170913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.171076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170916 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.171076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.170944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.172763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.172739 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.173191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.173167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.173379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.173363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.181409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.181389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxppk\" (UniqueName: \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\") pod \"maas-api-revoke-sub-86706870-68xc4\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.185904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.185881 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgn9h\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.186203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.186184 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.193109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.193079 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc042a5-0e5e-455f-8622-cf662acc6ead" path="/var/lib/kubelet/pods/7cc042a5-0e5e-455f-8622-cf662acc6ead/volumes" Oct 30 22:03:24.251503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.251472 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:24.321688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.321660 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:24.563619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.563576 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:03:24.582379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.568083 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:24.582379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.568184 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:24.582379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.568253 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:24.693773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.693653 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4"] Oct 30 22:03:24.696891 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:24.696862 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode275ca73_2729_4189_9572_cfdd6dc565a7.slice/crio-5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8 WatchSource:0}: Error finding container 5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8: Status 404 returned error can't find the container with id 5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8 Oct 30 22:03:24.916293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.915999 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" event={"ID":"59f2deb7-9007-4a20-9931-319aad1e0c2d","Type":"ContainerStarted","Data":"ad54f10af3fa220ed666a32f5de071f75fc43f6bfd6b2f5252ad1764dee2e345"} Oct 30 22:03:24.916293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.916071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" event={"ID":"59f2deb7-9007-4a20-9931-319aad1e0c2d","Type":"ContainerStarted","Data":"697e7334b08c1e7d1d02b2ee33197100f62ddc72c804b4e10db061ead37d0217"} Oct 30 22:03:24.943556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.943516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" event={"ID":"e275ca73-2729-4189-9572-cfdd6dc565a7","Type":"ContainerStarted","Data":"5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8"} Oct 30 22:03:24.962606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:24.962541 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podStartSLOduration=1.962518653 podStartE2EDuration="1.962518653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:24.959063452 +0000 UTC m=+1428.909018436" watchObservedRunningTime="2026-10-30 22:03:24.962518653 +0000 UTC m=+1428.912473601" Oct 30 22:03:25.252749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:25.252660 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:25.254191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:25.254158 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:25.254348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:25.254213 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:25.954699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:25.954660 5036 generic.go:356] "Generic (PLEG): container finished" podID="e275ca73-2729-4189-9572-cfdd6dc565a7" containerID="2f4a449518473bc7a6e60e5ea14ba9c6a0b3fe6527af7a69e4968d2a7a90b777" exitCode=0 Oct 30 22:03:25.955118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:25.954742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" event={"ID":"e275ca73-2729-4189-9572-cfdd6dc565a7","Type":"ContainerDied","Data":"2f4a449518473bc7a6e60e5ea14ba9c6a0b3fe6527af7a69e4968d2a7a90b777"} Oct 30 22:03:26.251896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.251854 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:26.252090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.251926 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:26.662605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.662572 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp"] Oct 30 22:03:26.666985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.666966 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.677579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.677550 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp"] Oct 30 22:03:26.763325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.763251 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.763596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.763404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.763596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.763532 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj29j\" (UniqueName: \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.864811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.864770 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.865022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.864896 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.865102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.865033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj29j\" (UniqueName: \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.865658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.865620 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.868561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.868523 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.877900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.877871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj29j\" (UniqueName: \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\") pod \"maas-api-revoke-sub-9fc26448-qrqbp\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:26.987490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:26.987435 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:27.200755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.200717 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:27.252589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.252504 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:27.252877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.252834 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:27.273229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.273194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\") pod \"e275ca73-2729-4189-9572-cfdd6dc565a7\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " Oct 30 22:03:27.273483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.273462 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\" (UniqueName: \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\") pod \"e275ca73-2729-4189-9572-cfdd6dc565a7\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " Oct 30 22:03:27.273820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.273760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\") pod \"e275ca73-2729-4189-9572-cfdd6dc565a7\" (UID: \"e275ca73-2729-4189-9572-cfdd6dc565a7\") " Oct 30 22:03:27.276311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.276283 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca" pod "e275ca73-2729-4189-9572-cfdd6dc565a7" (UID: "e275ca73-2729-4189-9572-cfdd6dc565a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:27.277241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.277218 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk" pod "e275ca73-2729-4189-9572-cfdd6dc565a7" (UID: "e275ca73-2729-4189-9572-cfdd6dc565a7"). InnerVolumeSpecName "kube-api-access-mxppk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:27.279411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.279350 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca" pod "e275ca73-2729-4189-9572-cfdd6dc565a7" (UID: "e275ca73-2729-4189-9572-cfdd6dc565a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:27.386546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.386462 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:27.386546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.386499 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e275ca73-2729-4189-9572-cfdd6dc565a7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:27.386546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.386515 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxppk\" (UniqueName: \"kubernetes.io/projected/e275ca73-2729-4189-9572-cfdd6dc565a7-kube-api-access-mxppk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:27.595044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.594994 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp"] Oct 30 22:03:27.975180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.975149 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" Oct 30 22:03:27.975180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.975166 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4" event={"ID":"e275ca73-2729-4189-9572-cfdd6dc565a7","Type":"ContainerDied","Data":"5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8"} Oct 30 22:03:27.975569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.975215 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5969d7b2613ed6811e4d7f368bd420fb49e1660eec2ed84e857e44f9666131d8" Oct 30 22:03:27.976774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:27.976749 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" event={"ID":"921f2b44-6917-481c-8a63-6d494c880b8b","Type":"ContainerStarted","Data":"9c42d1771b7b5b972f04f4e434fbd169eb921074887ddba790586c84b7c3bda4"} Oct 30 22:03:28.265295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.261092 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:28.265295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.261152 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:28.567348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.567223 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:03:28.568913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.568885 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e275ca73-2729-4189-9572-cfdd6dc565a7" containerName="revoke-keys" Oct 30 22:03:28.568913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.568912 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e275ca73-2729-4189-9572-cfdd6dc565a7" containerName="revoke-keys" Oct 30 22:03:28.569084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.569071 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e275ca73-2729-4189-9572-cfdd6dc565a7" containerName="revoke-keys" Oct 30 22:03:28.573418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.573401 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.584502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.584366 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2cfa71\"" Oct 30 22:03:28.586054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.586035 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2cfa71-dockercfg-2f8j5\"" Oct 30 22:03:28.590659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.590639 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:03:28.681291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.667114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.681291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.667380 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8qk\" (UniqueName: \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.770290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.769417 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.770290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.769522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8qk\" (UniqueName: \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.792288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.770801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.802747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.796970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8qk\" (UniqueName: \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\") pod \"payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:28.989638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:28.987489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:29.008298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:29.004281 5036 generic.go:356] "Generic (PLEG): container finished" podID="921f2b44-6917-481c-8a63-6d494c880b8b" containerID="3c588f4568cc3bb8ea33d3efa3838d0ad8d4de6e634842c1cd26ad57d2190e6f" exitCode=0 Oct 30 22:03:29.008298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:29.004587 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" event={"ID":"921f2b44-6917-481c-8a63-6d494c880b8b","Type":"ContainerDied","Data":"3c588f4568cc3bb8ea33d3efa3838d0ad8d4de6e634842c1cd26ad57d2190e6f"} Oct 30 22:03:29.291288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:29.282327 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:29.291288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:29.282410 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:29.773375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:29.773348 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:03:30.050342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.050230 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" event={"ID":"29559cc5-4272-495d-ac4a-fca1ac5cd192","Type":"ContainerStarted","Data":"0851017d5c5e3337563887c0701b94164468dcb409d340444e3ff814ddc9a263"} Oct 30 22:03:30.050342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.050312 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" event={"ID":"29559cc5-4272-495d-ac4a-fca1ac5cd192","Type":"ContainerStarted","Data":"b33c5d47b36bd5595f3bf7516f12644a92d601e662092b051a724f964f538324"} Oct 30 22:03:30.050599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.050383 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:30.070398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.070324 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" podStartSLOduration=2.070309532 podStartE2EDuration="2.070309532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:30.066976699 +0000 UTC m=+1434.016931645" watchObservedRunningTime="2026-10-30 22:03:30.070309532 +0000 UTC m=+1434.020264511" Oct 30 22:03:30.261289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.252572 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:30.261289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.252636 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:30.361651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.361623 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:30.490913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.490874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\") pod \"921f2b44-6917-481c-8a63-6d494c880b8b\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " Oct 30 22:03:30.491103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.490966 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\") pod \"921f2b44-6917-481c-8a63-6d494c880b8b\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " Oct 30 22:03:30.491103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.491018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\" (UniqueName: \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\") pod \"921f2b44-6917-481c-8a63-6d494c880b8b\" (UID: \"921f2b44-6917-481c-8a63-6d494c880b8b\") " Oct 30 22:03:30.491346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.491318 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca" pod "921f2b44-6917-481c-8a63-6d494c880b8b" (UID: "921f2b44-6917-481c-8a63-6d494c880b8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:30.491704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.491684 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:30.493594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.493569 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j" pod "921f2b44-6917-481c-8a63-6d494c880b8b" (UID: "921f2b44-6917-481c-8a63-6d494c880b8b"). InnerVolumeSpecName "kube-api-access-lj29j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:30.493814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.493793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca" pod "921f2b44-6917-481c-8a63-6d494c880b8b" (UID: "921f2b44-6917-481c-8a63-6d494c880b8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:30.593028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.592954 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f2b44-6917-481c-8a63-6d494c880b8b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:30.593028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.592982 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj29j\" (UniqueName: \"kubernetes.io/projected/921f2b44-6917-481c-8a63-6d494c880b8b-kube-api-access-lj29j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:30.758476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.758427 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr"] Oct 30 22:03:30.764531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:30.764506 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-90b6f5-29dsr"] Oct 30 22:03:31.063512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:31.063476 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" Oct 30 22:03:31.063512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:31.063490 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp" event={"ID":"921f2b44-6917-481c-8a63-6d494c880b8b","Type":"ContainerDied","Data":"9c42d1771b7b5b972f04f4e434fbd169eb921074887ddba790586c84b7c3bda4"} Oct 30 22:03:31.063747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:31.063537 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c42d1771b7b5b972f04f4e434fbd169eb921074887ddba790586c84b7c3bda4" Oct 30 22:03:31.261304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:31.260733 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 22:03:31.261304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:31.260808 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 22:03:32.196171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:32.196134 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2cf94b-ae2a-4617-98d6-50bc23ad2851" path="/var/lib/kubelet/pods/da2cf94b-ae2a-4617-98d6-50bc23ad2851/volumes" Oct 30 22:03:32.298826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:32.298779 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:32.299589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:32.299558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:32.300189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:32.300172 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:03:34.011258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.011226 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv"] Oct 30 22:03:34.013351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.013326 5036 cpu_manager.go:414] "containerMap: removing container" podUID="921f2b44-6917-481c-8a63-6d494c880b8b" containerName="revoke-keys" Oct 30 22:03:34.013491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.013353 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921f2b44-6917-481c-8a63-6d494c880b8b" containerName="revoke-keys" Oct 30 22:03:34.013594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.013582 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="921f2b44-6917-481c-8a63-6d494c880b8b" containerName="revoke-keys" Oct 30 22:03:34.018985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.018961 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.024149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.024127 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:34.024412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.024395 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:34.024596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.024580 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:34.024698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.024647 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:34.027746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.027708 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv"] Oct 30 22:03:34.106745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.106697 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.106917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.106849 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc84l\" (UniqueName: \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.106983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.106966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.208781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.208737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.208996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.208925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.209072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.209056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc84l\" (UniqueName: \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.209734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.209710 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.211694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.211672 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.221148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.221119 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc84l\" (UniqueName: \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\") pod \"maas-api-revoke-sub-7e6e1577-gfksv\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.345116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.345029 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:34.829405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:34.829373 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv"] Oct 30 22:03:34.829622 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:34.829581 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c972105_5945_44d6_81d4_abe096bbbd16.slice/crio-d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8 WatchSource:0}: Error finding container d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8: Status 404 returned error can't find the container with id d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8 Oct 30 22:03:35.126082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:35.125991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" event={"ID":"4c972105-5945-44d6-81d4-abe096bbbd16","Type":"ContainerStarted","Data":"d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8"} Oct 30 22:03:36.238831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:36.238728 5036 generic.go:356] "Generic (PLEG): container finished" podID="4c972105-5945-44d6-81d4-abe096bbbd16" containerID="1ac73af5297a033f835f708d0cd973495b0ce9b693b31fa5a3cdf629e1c4cf29" exitCode=0 Oct 30 22:03:36.262300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:36.261993 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" event={"ID":"4c972105-5945-44d6-81d4-abe096bbbd16","Type":"ContainerDied","Data":"1ac73af5297a033f835f708d0cd973495b0ce9b693b31fa5a3cdf629e1c4cf29"} Oct 30 22:03:37.160292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.159435 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.397430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.397377 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.412670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.412572 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.444419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.444372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.631281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.631215 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:03:37.928297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:37.925405 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.203420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.179624 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.203420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.180553 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:03:38.310323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.305134 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.398841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.398786 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.539850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.539748 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:03:38.567416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.563401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.647320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.631493 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:38.647320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.638814 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.669890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.669782 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.930129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:38.930075 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.140790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.137130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:03:39.140790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.137187 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:03:39.140790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.137227 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:03:39.140790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.137281 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:03:39.173292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.160413 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:03:39.181526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.181429 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.241782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.241736 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.351301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.318842 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.351301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.320341 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:03:39.412686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.412646 5036 scope.go:122] "RemoveContainer" containerID="a4450e0839b4634c53cee239726dfe61a4958f5e3c948178a2bfecc5dc80388c" Oct 30 22:03:39.501450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.416015 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:03:39.505532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.479012 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:03:39.578530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.566699 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.673363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.671672 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:39.675365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.673954 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.820477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.820360 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:39.850316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.843789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\") pod \"4c972105-5945-44d6-81d4-abe096bbbd16\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " Oct 30 22:03:39.854306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.850692 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\") pod \"4c972105-5945-44d6-81d4-abe096bbbd16\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " Oct 30 22:03:39.854722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.854695 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\" (UniqueName: \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\") pod \"4c972105-5945-44d6-81d4-abe096bbbd16\" (UID: \"4c972105-5945-44d6-81d4-abe096bbbd16\") " Oct 30 22:03:39.867314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.857828 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca" pod "4c972105-5945-44d6-81d4-abe096bbbd16" (UID: "4c972105-5945-44d6-81d4-abe096bbbd16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:39.887909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.887880 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:03:39.989484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.989443 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l" pod "4c972105-5945-44d6-81d4-abe096bbbd16" (UID: "4c972105-5945-44d6-81d4-abe096bbbd16"). InnerVolumeSpecName "kube-api-access-lc84l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:40.002106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.996483 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:03:40.002106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:39.998519 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.192381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.192069 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc84l\" (UniqueName: \"kubernetes.io/projected/4c972105-5945-44d6-81d4-abe096bbbd16-kube-api-access-lc84l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.284657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.277944 5036 scope.go:122] "RemoveContainer" containerID="bf9addbc0d544a95544f4edd08c490f23261f8bd04491c5dff3328e6c5a9a86c" Oct 30 22:03:40.293984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.287521 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:40.332699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.332672 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:03:40.355289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.344565 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca" pod "4c972105-5945-44d6-81d4-abe096bbbd16" (UID: "4c972105-5945-44d6-81d4-abe096bbbd16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:40.355496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.344653 5036 scope.go:122] "RemoveContainer" containerID="fcb948ba0c3334f754575ce9e74be8e6131579519dc78587526e758dae91a556" Oct 30 22:03:40.373159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.371847 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:03:40.388594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.388569 5036 scope.go:122] "RemoveContainer" containerID="ac38e53afb942c73817c4b2612fcb01658ac08f6f6f0ce8094ffcd9230b9974a" Oct 30 22:03:40.425724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.425183 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c972105-5945-44d6-81d4-abe096bbbd16-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.446287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.442891 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:03:40.446287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.443911 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:03:40.581036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.581012 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" Oct 30 22:03:40.581143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.581033 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv" event={"ID":"4c972105-5945-44d6-81d4-abe096bbbd16","Type":"ContainerDied","Data":"d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8"} Oct 30 22:03:40.581143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:40.581113 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8251a7e835979e91b7be6056e3ddea9f8198ee95872d9428f82ccfda9918bd8" Oct 30 22:03:41.951750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.951711 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl"] Oct 30 22:03:41.953414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.953352 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4c972105-5945-44d6-81d4-abe096bbbd16" containerName="revoke-keys" Oct 30 22:03:41.953654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.953622 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c972105-5945-44d6-81d4-abe096bbbd16" containerName="revoke-keys" Oct 30 22:03:41.954052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.954020 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c972105-5945-44d6-81d4-abe096bbbd16" containerName="revoke-keys" Oct 30 22:03:41.967837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.967808 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl"] Oct 30 22:03:41.967990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.967931 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:41.975042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.975015 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:41.975227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.975153 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:41.975227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.975172 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:41.975808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:41.975783 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:42.076988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.076950 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktq6\" (UniqueName: \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.077195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.077035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.077195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.077181 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.079445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.079423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:03:42.178363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.178327 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.178580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.178553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktq6\" (UniqueName: \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.178653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.178585 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.179035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.179013 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.181202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.181178 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.192228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.192200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktq6\" (UniqueName: \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\") pod \"maas-api-revoke-sub-e322c572-f6ntl\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.284965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.284873 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:42.296818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.296789 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc"] Oct 30 22:03:42.332312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.328607 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.368426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.368388 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc"] Oct 30 22:03:42.490184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.489456 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pp6\" (UniqueName: \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.490184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.489546 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.490184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.489591 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.590634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.590539 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pp6\" (UniqueName: \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.590634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.590592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.590634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.590618 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.591328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.591300 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.593332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.593309 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.602203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.602182 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pp6\" (UniqueName: \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\") pod \"maas-api-revoke-sub-b95e2146-h2rnc\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:42.678776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.678745 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl"] Oct 30 22:03:42.679351 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:42.679327 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bc0838_3a30_4647_8da6_1fa7b2199f14.slice/crio-06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94 WatchSource:0}: Error finding container 06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94: Status 404 returned error can't find the container with id 06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94 Oct 30 22:03:42.701834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:42.701815 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:43.055743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.055706 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc"] Oct 30 22:03:43.637970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.637927 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" event={"ID":"530b98ec-ba52-4908-bfac-8016bb83cf5f","Type":"ContainerStarted","Data":"0110b48d40c09c417a311855c7bd7bacfeba9f78216dae543d92be718ae474f6"} Oct 30 22:03:43.642352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.642311 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" event={"ID":"530b98ec-ba52-4908-bfac-8016bb83cf5f","Type":"ContainerStarted","Data":"1773881403bbf5d37408d5cfb41284cb6821454f842de0acc43bd4f9269cc9a2"} Oct 30 22:03:43.653651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.653618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" event={"ID":"29bc0838-3a30-4647-8da6-1fa7b2199f14","Type":"ContainerStarted","Data":"a94efe1b61d3b188dd2d723bb9a1d2915edac2691dd0bb29f0bd2ea93cd58b90"} Oct 30 22:03:43.654619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.654588 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" event={"ID":"29bc0838-3a30-4647-8da6-1fa7b2199f14","Type":"ContainerStarted","Data":"06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94"} Oct 30 22:03:43.671783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.671723 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" podStartSLOduration=1.671706975 podStartE2EDuration="1.671706975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:43.664374038 +0000 UTC m=+1447.614328988" watchObservedRunningTime="2026-10-30 22:03:43.671706975 +0000 UTC m=+1447.621661922" Oct 30 22:03:43.685858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:43.685816 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" podStartSLOduration=2.685803937 podStartE2EDuration="2.685803937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:43.682411503 +0000 UTC m=+1447.632366457" watchObservedRunningTime="2026-10-30 22:03:43.685803937 +0000 UTC m=+1447.635758907" Oct 30 22:03:44.664105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:44.664073 5036 generic.go:356] "Generic (PLEG): container finished" podID="530b98ec-ba52-4908-bfac-8016bb83cf5f" containerID="0110b48d40c09c417a311855c7bd7bacfeba9f78216dae543d92be718ae474f6" exitCode=0 Oct 30 22:03:44.664611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:44.664155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" event={"ID":"530b98ec-ba52-4908-bfac-8016bb83cf5f","Type":"ContainerDied","Data":"0110b48d40c09c417a311855c7bd7bacfeba9f78216dae543d92be718ae474f6"} Oct 30 22:03:44.666229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:44.666205 5036 generic.go:356] "Generic (PLEG): container finished" podID="29bc0838-3a30-4647-8da6-1fa7b2199f14" containerID="a94efe1b61d3b188dd2d723bb9a1d2915edac2691dd0bb29f0bd2ea93cd58b90" exitCode=0 Oct 30 22:03:44.666357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:44.666294 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" event={"ID":"29bc0838-3a30-4647-8da6-1fa7b2199f14","Type":"ContainerDied","Data":"a94efe1b61d3b188dd2d723bb9a1d2915edac2691dd0bb29f0bd2ea93cd58b90"} Oct 30 22:03:45.486122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.486061 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm"] Oct 30 22:03:45.491024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.491000 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.501302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.501254 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm"] Oct 30 22:03:45.538236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.538198 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.538473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.538296 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dnc\" (UniqueName: \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.538473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.538402 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.640503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.640458 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.640695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.640527 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dnc\" (UniqueName: \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.640695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.640548 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.641198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.641169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.643094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.643071 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.673912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.673872 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dnc\" (UniqueName: \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\") pod \"maas-api-revoke-sub-1d394678-bfnnm\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.816904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.816865 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:45.939862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.938148 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:45.951686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:45.951595 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:46.041782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.041420 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:46.041782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.041481 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048505 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\") pod \"29bc0838-3a30-4647-8da6-1fa7b2199f14\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048544 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\") pod \"530b98ec-ba52-4908-bfac-8016bb83cf5f\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048694 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\" (UniqueName: \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\") pod \"530b98ec-ba52-4908-bfac-8016bb83cf5f\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048733 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\") pod \"29bc0838-3a30-4647-8da6-1fa7b2199f14\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048759 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\" (UniqueName: \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\") pod \"29bc0838-3a30-4647-8da6-1fa7b2199f14\" (UID: \"29bc0838-3a30-4647-8da6-1fa7b2199f14\") " Oct 30 22:03:46.049445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.048784 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\") pod \"530b98ec-ba52-4908-bfac-8016bb83cf5f\" (UID: \"530b98ec-ba52-4908-bfac-8016bb83cf5f\") " Oct 30 22:03:46.051750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.051601 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca" pod "29bc0838-3a30-4647-8da6-1fa7b2199f14" (UID: "29bc0838-3a30-4647-8da6-1fa7b2199f14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:46.051750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.051466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca" pod "530b98ec-ba52-4908-bfac-8016bb83cf5f" (UID: "530b98ec-ba52-4908-bfac-8016bb83cf5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:46.054710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.054674 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca" pod "29bc0838-3a30-4647-8da6-1fa7b2199f14" (UID: "29bc0838-3a30-4647-8da6-1fa7b2199f14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:46.054806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.054749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6" pod "29bc0838-3a30-4647-8da6-1fa7b2199f14" (UID: "29bc0838-3a30-4647-8da6-1fa7b2199f14"). InnerVolumeSpecName "kube-api-access-pktq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:46.058132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.058103 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6" pod "530b98ec-ba52-4908-bfac-8016bb83cf5f" (UID: "530b98ec-ba52-4908-bfac-8016bb83cf5f"). InnerVolumeSpecName "kube-api-access-w8pp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:46.060011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.059978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca" pod "530b98ec-ba52-4908-bfac-8016bb83cf5f" (UID: "530b98ec-ba52-4908-bfac-8016bb83cf5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:46.151162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151126 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.151162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151155 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.151162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151166 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pp6\" (UniqueName: \"kubernetes.io/projected/530b98ec-ba52-4908-bfac-8016bb83cf5f-kube-api-access-w8pp6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.151523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151177 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0838-3a30-4647-8da6-1fa7b2199f14-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.151523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151186 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktq6\" (UniqueName: \"kubernetes.io/projected/29bc0838-3a30-4647-8da6-1fa7b2199f14-kube-api-access-pktq6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.151523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.151195 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530b98ec-ba52-4908-bfac-8016bb83cf5f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:46.243549 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:46.243510 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717fd2a1_8a3c_48d2_ba92_8dd42fd295bb.slice/crio-80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e WatchSource:0}: Error finding container 80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e: Status 404 returned error can't find the container with id 80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e Oct 30 22:03:46.243778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.243761 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm"] Oct 30 22:03:46.689689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.689654 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" Oct 30 22:03:46.689689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.689664 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl" event={"ID":"29bc0838-3a30-4647-8da6-1fa7b2199f14","Type":"ContainerDied","Data":"06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94"} Oct 30 22:03:46.690242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.689745 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b4fcee50809961d0d92c7f60d7bcab6c88fed23136c61887a224937847bf94" Oct 30 22:03:46.691626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.691594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" event={"ID":"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb","Type":"ContainerStarted","Data":"ee2356ab6186a8339d5a7e97975599d660c5d4faace7136ef00af29346c2fadc"} Oct 30 22:03:46.691766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.691652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" event={"ID":"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb","Type":"ContainerStarted","Data":"80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e"} Oct 30 22:03:46.693527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.693501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" event={"ID":"530b98ec-ba52-4908-bfac-8016bb83cf5f","Type":"ContainerDied","Data":"1773881403bbf5d37408d5cfb41284cb6821454f842de0acc43bd4f9269cc9a2"} Oct 30 22:03:46.693608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.693542 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc" Oct 30 22:03:46.693608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.693551 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1773881403bbf5d37408d5cfb41284cb6821454f842de0acc43bd4f9269cc9a2" Oct 30 22:03:46.710530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:46.710479 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" podStartSLOduration=1.7104685750000002 podStartE2EDuration="1.710468575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:46.710024883 +0000 UTC m=+1450.659979830" watchObservedRunningTime="2026-10-30 22:03:46.710468575 +0000 UTC m=+1450.660423543" Oct 30 22:03:47.705910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:47.705871 5036 generic.go:356] "Generic (PLEG): container finished" podID="717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" containerID="ee2356ab6186a8339d5a7e97975599d660c5d4faace7136ef00af29346c2fadc" exitCode=0 Oct 30 22:03:47.706427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:47.705974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" event={"ID":"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb","Type":"ContainerDied","Data":"ee2356ab6186a8339d5a7e97975599d660c5d4faace7136ef00af29346c2fadc"} Oct 30 22:03:48.103503 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:03:48.103405 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:03:48.718309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.718235 5036 generic.go:356] "Generic (PLEG): container finished" podID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerID="17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202" exitCode=0 Oct 30 22:03:48.718813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.718332 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerDied","Data":"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202"} Oct 30 22:03:48.719191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.718913 5036 scope.go:122] "RemoveContainer" containerID="17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202" Oct 30 22:03:48.720413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.720374 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:48.720581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.720466 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:48.720581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:48.720503 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:49.032579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.032550 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:49.126466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.126429 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\") pod \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " Oct 30 22:03:49.126642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.126524 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\") pod \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " Oct 30 22:03:49.126642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.126558 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\" (UniqueName: \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\") pod \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\" (UID: \"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb\") " Oct 30 22:03:49.126895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.126865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca" pod "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" (UID: "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:49.127148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.127129 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.128937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.128908 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca" pod "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" (UID: "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:49.129248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.129229 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc" pod "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" (UID: "717fd2a1-8a3c-48d2-ba92-8dd42fd295bb"). InnerVolumeSpecName "kube-api-access-w4dnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:49.228308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.228250 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dnc\" (UniqueName: \"kubernetes.io/projected/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-kube-api-access-w4dnc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.228308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.228310 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.754277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.754233 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" Oct 30 22:03:49.754928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.754303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm" event={"ID":"717fd2a1-8a3c-48d2-ba92-8dd42fd295bb","Type":"ContainerDied","Data":"80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e"} Oct 30 22:03:49.755028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.754980 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80212913732a849c89f8fb164407acc5cc915564ec11752c7ca7000a25728d9e" Oct 30 22:03:49.757952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:49.757923 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerStarted","Data":"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4"} Oct 30 22:03:50.140349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:50.131130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:03:50.152289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:50.143404 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 22:03:50.152289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:50.143483 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 22:03:51.129289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:51.128688 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 22:03:51.129289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:51.128766 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 22:03:52.134290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:52.133697 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 22:03:52.134290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:52.133757 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 22:03:53.129300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:53.128653 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 22:03:53.129300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:53.128725 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 22:03:54.133931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.133886 5036 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 22:03:54.134531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.133967 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 22:03:54.157859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.157830 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42"] Oct 30 22:03:54.158728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.158708 5036 cpu_manager.go:414] "containerMap: removing container" podUID="530b98ec-ba52-4908-bfac-8016bb83cf5f" containerName="revoke-keys" Oct 30 22:03:54.158845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.158835 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530b98ec-ba52-4908-bfac-8016bb83cf5f" containerName="revoke-keys" Oct 30 22:03:54.158921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.158913 5036 cpu_manager.go:414] "containerMap: removing container" podUID="717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" containerName="revoke-keys" Oct 30 22:03:54.158981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.158974 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" containerName="revoke-keys" Oct 30 22:03:54.159043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.159037 5036 cpu_manager.go:414] "containerMap: removing container" podUID="29bc0838-3a30-4647-8da6-1fa7b2199f14" containerName="revoke-keys" Oct 30 22:03:54.159092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.159085 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bc0838-3a30-4647-8da6-1fa7b2199f14" containerName="revoke-keys" Oct 30 22:03:54.159322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.159312 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="530b98ec-ba52-4908-bfac-8016bb83cf5f" containerName="revoke-keys" Oct 30 22:03:54.159410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.159402 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" containerName="revoke-keys" Oct 30 22:03:54.159471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.159464 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bc0838-3a30-4647-8da6-1fa7b2199f14" containerName="revoke-keys" Oct 30 22:03:54.163257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.163240 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.167359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.167329 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:54.167679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.167661 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:54.167958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.167936 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:03:54.168077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.168006 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:54.170919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.170891 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42"] Oct 30 22:03:54.241581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.241112 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.241581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.241295 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.241581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.241335 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.343226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.343194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.343438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.343308 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.343438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.343341 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.344026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.344006 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.346413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.346390 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.355435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.355389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\") pod \"maas-api-revoke-sub-dbd06d2e-46r42\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.492292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.492202 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:03:54.816689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:54.816660 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42"] Oct 30 22:03:54.819025 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:03:54.818997 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a00178_7496_47fb_96a0_1bfcef277fa3.slice/crio-8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d WatchSource:0}: Error finding container 8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d: Status 404 returned error can't find the container with id 8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d Oct 30 22:03:55.133469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.133439 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:03:55.133857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.133840 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:03:55.134642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.134621 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:03:55.813512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.813413 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" event={"ID":"01a00178-7496-47fb-96a0-1bfcef277fa3","Type":"ContainerStarted","Data":"a4374d19ebf9701ae70c4a9b37656ec6476d6faefacdd53831b79c1d4e0d9097"} Oct 30 22:03:55.813512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.813475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" event={"ID":"01a00178-7496-47fb-96a0-1bfcef277fa3","Type":"ContainerStarted","Data":"8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d"} Oct 30 22:03:55.840014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:55.839959 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" podStartSLOduration=1.839941677 podStartE2EDuration="1.839941677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:55.836107526 +0000 UTC m=+1459.786062472" watchObservedRunningTime="2026-10-30 22:03:55.839941677 +0000 UTC m=+1459.789896635" Oct 30 22:03:56.825568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:56.825525 5036 generic.go:356] "Generic (PLEG): container finished" podID="01a00178-7496-47fb-96a0-1bfcef277fa3" containerID="a4374d19ebf9701ae70c4a9b37656ec6476d6faefacdd53831b79c1d4e0d9097" exitCode=0 Oct 30 22:03:56.826003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:56.825955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" event={"ID":"01a00178-7496-47fb-96a0-1bfcef277fa3","Type":"ContainerDied","Data":"a4374d19ebf9701ae70c4a9b37656ec6476d6faefacdd53831b79c1d4e0d9097"} Oct 30 22:03:57.776726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.773318 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph"] Oct 30 22:03:57.788866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.778958 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:57.880287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.875504 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:57.880287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.875611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:57.880287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.875717 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmrf\" (UniqueName: \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:57.880287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:57.875514 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph"] Oct 30 22:03:58.103352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.076744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.103352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.076844 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.103352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.076916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmrf\" (UniqueName: \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.126569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.104730 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.414360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.410991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.448195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.448151 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmrf\" (UniqueName: \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\") pod \"maas-api-revoke-sub-1b563f2a-2vdph\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:58.726557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:58.726519 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:03:59.082623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.081371 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:03:59.089315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.087817 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.209668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.208388 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.209668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.208846 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 22:03:59.322693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.322570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.764604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.760852 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.764604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.761310 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.764604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.761368 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:03:59.766492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:03:59.766462 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:04:00.093835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.087379 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.093835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.088686 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:04:00.266450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.205991 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 30 22:04:00.268867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.267622 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.268867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.268436 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:00.305293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.304389 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.369062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.362304 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.806367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.806313 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.833868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.833814 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.834211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.834185 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.365366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.365302 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:04:01.406295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.373253 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:01.486014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.988473 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:04:01.486014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.195209 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:04:01.497918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:00.884728 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:04:01.642429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.625833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") pod \"01a00178-7496-47fb-96a0-1bfcef277fa3\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " Oct 30 22:04:01.642429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.626050 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\") pod \"01a00178-7496-47fb-96a0-1bfcef277fa3\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " Oct 30 22:04:01.642429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.626190 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\") pod \"01a00178-7496-47fb-96a0-1bfcef277fa3\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.097546 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.656442 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph"] Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.656612 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.656679 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.656740 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:01.664304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.656801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:01.718295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.684419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" event={"ID":"01a00178-7496-47fb-96a0-1bfcef277fa3","Type":"ContainerDied","Data":"8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d"} Oct 30 22:04:01.718295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.684543 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c98dde8231813e463f35f3f7c8ec0b3900f1214abb24f23e2dd9abec6ecdb3d" Oct 30 22:04:01.892350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.866861 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:01.892350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.867112 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.925289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.905878 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:01.994296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:01.969752 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.240361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.234464 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:02.240361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.238342 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.273339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.259866 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j" pod "01a00178-7496-47fb-96a0-1bfcef277fa3" (UID: "01a00178-7496-47fb-96a0-1bfcef277fa3"). InnerVolumeSpecName "kube-api-access-rnp9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:02.273578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.270158 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca" pod "01a00178-7496-47fb-96a0-1bfcef277fa3" (UID: "01a00178-7496-47fb-96a0-1bfcef277fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:02.370663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.370608 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca" pod "01a00178-7496-47fb-96a0-1bfcef277fa3" (UID: "01a00178-7496-47fb-96a0-1bfcef277fa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:02.486296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.485443 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:02.486296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.485515 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.486896 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") pod \"01a00178-7496-47fb-96a0-1bfcef277fa3\" (UID: \"01a00178-7496-47fb-96a0-1bfcef277fa3\") " Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.487823 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.487849 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/01a00178-7496-47fb-96a0-1bfcef277fa3-kube-api-access-rnp9j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:02.487944 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/01a00178-7496-47fb-96a0-1bfcef277fa3/volumes/kubernetes.io~configmap/maas-api-openshift-ca Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.487959 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca" pod "01a00178-7496-47fb-96a0-1bfcef277fa3" (UID: "01a00178-7496-47fb-96a0-1bfcef277fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:02.503344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.370424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.515541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.515502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.617594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.617538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:04:02.624298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.617919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:04:02.660338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.650176 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a00178-7496-47fb-96a0-1bfcef277fa3-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.660338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.650246 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:02.660338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.650314 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.947717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.947673 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:02.995426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:02.995383 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:04:03.303345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.285921 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42" Oct 30 22:04:03.303345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.294658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" event={"ID":"e4fdf083-ed02-4a9c-9588-e33cb2e134b9","Type":"ContainerStarted","Data":"2f5c8f5db437975c859cc4f0d6863eab88ff832e4cd4f98a32f50a2a0f0e03a4"} Oct 30 22:04:03.497417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.497368 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:04:03.537288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.524485 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:03.579289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.574703 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:03.606044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.604449 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:03.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.618373 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp"] Oct 30 22:04:03.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.619749 5036 cpu_manager.go:414] "containerMap: removing container" podUID="01a00178-7496-47fb-96a0-1bfcef277fa3" containerName="revoke-keys" Oct 30 22:04:03.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.619772 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a00178-7496-47fb-96a0-1bfcef277fa3" containerName="revoke-keys" Oct 30 22:04:03.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.619979 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a00178-7496-47fb-96a0-1bfcef277fa3" containerName="revoke-keys" Oct 30 22:04:03.621826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.620790 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:04:03.621826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.620789 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:03.652692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.652660 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp"] Oct 30 22:04:03.653026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.653006 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.674443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.674095 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:03.674443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.674159 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:03.747715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.718789 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cfn\" (UniqueName: \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.747715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.718913 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.747715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.719123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.758628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.758592 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:03.758896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.758868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:03.833340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.826121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.833340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.826283 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cfn\" (UniqueName: \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.833340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.826378 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.887847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.848648 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.887847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.875861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.901289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.899877 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:04:03.911788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.902046 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:04:03.911788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.902689 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:04:03.911788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.903305 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:03.911788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.907246 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cfn\" (UniqueName: \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\") pod \"maas-api-revoke-sub-daeb3c08-k9mjp\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:03.921633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.921603 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:03.923857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:03.923834 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:04.109302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:04.109180 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:04.312371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:04.312331 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" event={"ID":"e4fdf083-ed02-4a9c-9588-e33cb2e134b9","Type":"ContainerStarted","Data":"e110ed72b1f832b78f1d65de58c852e691a475bbb92441cdec9b0592885b7b96"} Oct 30 22:04:04.344756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:04.344692 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" podStartSLOduration=7.344672297 podStartE2EDuration="7.344672297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:04.338514685 +0000 UTC m=+1468.288469641" watchObservedRunningTime="2026-10-30 22:04:04.344672297 +0000 UTC m=+1468.294627244" Oct 30 22:04:04.416745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:04.416717 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp"] Oct 30 22:04:04.417212 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:04.417186 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ea7c9f_b080_400d_82d5_c7b52ed302e6.slice/crio-8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8 WatchSource:0}: Error finding container 8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8: Status 404 returned error can't find the container with id 8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8 Oct 30 22:04:05.323386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:05.323350 5036 generic.go:356] "Generic (PLEG): container finished" podID="e4fdf083-ed02-4a9c-9588-e33cb2e134b9" containerID="e110ed72b1f832b78f1d65de58c852e691a475bbb92441cdec9b0592885b7b96" exitCode=0 Oct 30 22:04:05.323797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:05.323432 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" event={"ID":"e4fdf083-ed02-4a9c-9588-e33cb2e134b9","Type":"ContainerDied","Data":"e110ed72b1f832b78f1d65de58c852e691a475bbb92441cdec9b0592885b7b96"} Oct 30 22:04:05.325500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:05.325456 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" event={"ID":"85ea7c9f-b080-400d-82d5-c7b52ed302e6","Type":"ContainerStarted","Data":"302153d49b702e0908b122eb9f88065a0079d20bafa88da3585e23a7ca5320c8"} Oct 30 22:04:05.325652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:05.325514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" event={"ID":"85ea7c9f-b080-400d-82d5-c7b52ed302e6","Type":"ContainerStarted","Data":"8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8"} Oct 30 22:04:05.387093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:05.387041 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" podStartSLOduration=3.387027469 podStartE2EDuration="3.387027469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:05.380082689 +0000 UTC m=+1469.330037763" watchObservedRunningTime="2026-10-30 22:04:05.387027469 +0000 UTC m=+1469.336982460" Oct 30 22:04:06.350641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.350599 5036 generic.go:356] "Generic (PLEG): container finished" podID="85ea7c9f-b080-400d-82d5-c7b52ed302e6" containerID="302153d49b702e0908b122eb9f88065a0079d20bafa88da3585e23a7ca5320c8" exitCode=0 Oct 30 22:04:06.351104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.350683 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" event={"ID":"85ea7c9f-b080-400d-82d5-c7b52ed302e6","Type":"ContainerDied","Data":"302153d49b702e0908b122eb9f88065a0079d20bafa88da3585e23a7ca5320c8"} Oct 30 22:04:06.369395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.369368 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8"] Oct 30 22:04:06.374233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.374108 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.388092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.387181 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8"] Oct 30 22:04:06.474731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.474699 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.474931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.474828 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.475042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.475022 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttdp\" (UniqueName: \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.548738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.548715 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:04:06.576244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.576212 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\") pod \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " Oct 30 22:04:06.576449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.576348 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\" (UniqueName: \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\") pod \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " Oct 30 22:04:06.576866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.576840 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\") pod \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\" (UID: \"e4fdf083-ed02-4a9c-9588-e33cb2e134b9\") " Oct 30 22:04:06.577490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.577460 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca" pod "e4fdf083-ed02-4a9c-9588-e33cb2e134b9" (UID: "e4fdf083-ed02-4a9c-9588-e33cb2e134b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:06.577605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.577474 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttdp\" (UniqueName: \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.577727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.577705 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.578259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.577815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.578259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.577990 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.580342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.580312 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.580480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.580313 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca" pod "e4fdf083-ed02-4a9c-9588-e33cb2e134b9" (UID: "e4fdf083-ed02-4a9c-9588-e33cb2e134b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:06.580480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.580391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf" pod "e4fdf083-ed02-4a9c-9588-e33cb2e134b9" (UID: "e4fdf083-ed02-4a9c-9588-e33cb2e134b9"). InnerVolumeSpecName "kube-api-access-brmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:06.581074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.581049 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.591232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.591213 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttdp\" (UniqueName: \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\") pod \"maas-api-revoke-sub-2040ee18-48rd8\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:06.679704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.679678 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.679704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.679706 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmrf\" (UniqueName: \"kubernetes.io/projected/e4fdf083-ed02-4a9c-9588-e33cb2e134b9-kube-api-access-brmrf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.701718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:06.701694 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:07.024533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.024505 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8"] Oct 30 22:04:07.025077 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:07.025049 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6af612b_f7ce_427a_b4a8_4d550a961a85.slice/crio-b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06 WatchSource:0}: Error finding container b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06: Status 404 returned error can't find the container with id b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06 Oct 30 22:04:07.361112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.361033 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" Oct 30 22:04:07.361112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.361066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph" event={"ID":"e4fdf083-ed02-4a9c-9588-e33cb2e134b9","Type":"ContainerDied","Data":"2f5c8f5db437975c859cc4f0d6863eab88ff832e4cd4f98a32f50a2a0f0e03a4"} Oct 30 22:04:07.361593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.361129 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5c8f5db437975c859cc4f0d6863eab88ff832e4cd4f98a32f50a2a0f0e03a4" Oct 30 22:04:07.363027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.362987 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" event={"ID":"e6af612b-f7ce-427a-b4a8-4d550a961a85","Type":"ContainerStarted","Data":"b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06"} Oct 30 22:04:07.611935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.611874 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:07.695620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.695593 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\") pod \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " Oct 30 22:04:07.695806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.695710 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\") pod \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " Oct 30 22:04:07.695806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.695767 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\" (UniqueName: \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\") pod \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\" (UID: \"85ea7c9f-b080-400d-82d5-c7b52ed302e6\") " Oct 30 22:04:07.696051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.696026 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca" pod "85ea7c9f-b080-400d-82d5-c7b52ed302e6" (UID: "85ea7c9f-b080-400d-82d5-c7b52ed302e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:07.696374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.696346 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.698022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.697999 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca" pod "85ea7c9f-b080-400d-82d5-c7b52ed302e6" (UID: "85ea7c9f-b080-400d-82d5-c7b52ed302e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:07.698218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.698196 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn" pod "85ea7c9f-b080-400d-82d5-c7b52ed302e6" (UID: "85ea7c9f-b080-400d-82d5-c7b52ed302e6"). InnerVolumeSpecName "kube-api-access-f4cfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:07.797422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.797397 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ea7c9f-b080-400d-82d5-c7b52ed302e6-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.797422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:07.797420 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cfn\" (UniqueName: \"kubernetes.io/projected/85ea7c9f-b080-400d-82d5-c7b52ed302e6-kube-api-access-f4cfn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:08.374492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:08.374410 5036 generic.go:356] "Generic (PLEG): container finished" podID="e6af612b-f7ce-427a-b4a8-4d550a961a85" containerID="0b8a4214b88975b91eee158cbf3907e79840b48391641b139517ccbcd83bbd3d" exitCode=0 Oct 30 22:04:08.374896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:08.374495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" event={"ID":"e6af612b-f7ce-427a-b4a8-4d550a961a85","Type":"ContainerDied","Data":"0b8a4214b88975b91eee158cbf3907e79840b48391641b139517ccbcd83bbd3d"} Oct 30 22:04:08.376174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:08.376155 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" Oct 30 22:04:08.376365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:08.376185 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp" event={"ID":"85ea7c9f-b080-400d-82d5-c7b52ed302e6","Type":"ContainerDied","Data":"8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8"} Oct 30 22:04:08.376365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:08.376225 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8311e389819bfa417aaa93300557ad33b6656bd2face602fbee577b52a60d7c8" Oct 30 22:04:09.556758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.556724 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:09.625286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.622110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\") pod \"e6af612b-f7ce-427a-b4a8-4d550a961a85\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " Oct 30 22:04:09.625286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.622343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\") pod \"e6af612b-f7ce-427a-b4a8-4d550a961a85\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " Oct 30 22:04:09.625286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.622489 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\" (UniqueName: \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\") pod \"e6af612b-f7ce-427a-b4a8-4d550a961a85\" (UID: \"e6af612b-f7ce-427a-b4a8-4d550a961a85\") " Oct 30 22:04:09.625286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.624697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca" pod "e6af612b-f7ce-427a-b4a8-4d550a961a85" (UID: "e6af612b-f7ce-427a-b4a8-4d550a961a85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:09.633469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.633440 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca" pod "e6af612b-f7ce-427a-b4a8-4d550a961a85" (UID: "e6af612b-f7ce-427a-b4a8-4d550a961a85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:09.633469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.633455 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp" pod "e6af612b-f7ce-427a-b4a8-4d550a961a85" (UID: "e6af612b-f7ce-427a-b4a8-4d550a961a85"). InnerVolumeSpecName "kube-api-access-bttdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.724992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.724957 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.724992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.724993 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6af612b-f7ce-427a-b4a8-4d550a961a85-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.725184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:09.725010 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttdp\" (UniqueName: \"kubernetes.io/projected/e6af612b-f7ce-427a-b4a8-4d550a961a85-kube-api-access-bttdp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.328941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.328906 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp"] Oct 30 22:04:10.329999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.329977 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e4fdf083-ed02-4a9c-9588-e33cb2e134b9" containerName="revoke-keys" Oct 30 22:04:10.330051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330003 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fdf083-ed02-4a9c-9588-e33cb2e134b9" containerName="revoke-keys" Oct 30 22:04:10.330051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330021 5036 cpu_manager.go:414] "containerMap: removing container" podUID="85ea7c9f-b080-400d-82d5-c7b52ed302e6" containerName="revoke-keys" Oct 30 22:04:10.330051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330028 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ea7c9f-b080-400d-82d5-c7b52ed302e6" containerName="revoke-keys" Oct 30 22:04:10.330173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330059 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e6af612b-f7ce-427a-b4a8-4d550a961a85" containerName="revoke-keys" Oct 30 22:04:10.330173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330067 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6af612b-f7ce-427a-b4a8-4d550a961a85" containerName="revoke-keys" Oct 30 22:04:10.330257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330245 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fdf083-ed02-4a9c-9588-e33cb2e134b9" containerName="revoke-keys" Oct 30 22:04:10.330313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330278 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ea7c9f-b080-400d-82d5-c7b52ed302e6" containerName="revoke-keys" Oct 30 22:04:10.330313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.330295 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6af612b-f7ce-427a-b4a8-4d550a961a85" containerName="revoke-keys" Oct 30 22:04:10.334894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.334869 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.341440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.341412 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp"] Oct 30 22:04:10.394170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.394140 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" Oct 30 22:04:10.394372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.394174 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8" event={"ID":"e6af612b-f7ce-427a-b4a8-4d550a961a85","Type":"ContainerDied","Data":"b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06"} Oct 30 22:04:10.394372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.394219 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19b003fedf87fbf42f40721517de98a30f4064ef223070e66fd4707517ebd06" Oct 30 22:04:10.435276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.435225 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.435448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.435280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzprt\" (UniqueName: \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.435492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.435450 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.536889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.536802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.536889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.536838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzprt\" (UniqueName: \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.537156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.536904 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.537680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.537652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.539652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.539628 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.548880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.548856 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzprt\" (UniqueName: \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\") pod \"maas-api-revoke-sub-234d5da1-qfbbp\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.656592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.656557 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:10.695516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.695482 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 22:04:10.695710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.695687 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" containerID="cri-o://373783f7c42b3ce58452dc9e0262220fa76871b5957ba6a75680593aa643063c" gracePeriod=30 Oct 30 22:04:10.959159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:10.959125 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp"] Oct 30 22:04:10.965862 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:10.965822 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4dbf5a_3c29_40cf_93cf_a3e5baeb36b5.slice/crio-27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0 WatchSource:0}: Error finding container 27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0: Status 404 returned error can't find the container with id 27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0 Oct 30 22:04:11.405800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:11.405759 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" event={"ID":"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5","Type":"ContainerStarted","Data":"f92d05cda3850cac331fe7774146b9b80882b4eec9e5600024c833395b97d8fd"} Oct 30 22:04:11.405983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:11.405865 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" event={"ID":"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5","Type":"ContainerStarted","Data":"27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0"} Oct 30 22:04:11.430949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:11.430891 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" podStartSLOduration=1.430877729 podStartE2EDuration="1.430877729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:11.426639377 +0000 UTC m=+1475.376594321" watchObservedRunningTime="2026-10-30 22:04:11.430877729 +0000 UTC m=+1475.380832675" Oct 30 22:04:12.418307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:12.418233 5036 generic.go:356] "Generic (PLEG): container finished" podID="ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" containerID="f92d05cda3850cac331fe7774146b9b80882b4eec9e5600024c833395b97d8fd" exitCode=0 Oct 30 22:04:12.418792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:12.418371 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" event={"ID":"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5","Type":"ContainerDied","Data":"f92d05cda3850cac331fe7774146b9b80882b4eec9e5600024c833395b97d8fd"} Oct 30 22:04:14.565524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.565483 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:14.625308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.621219 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\" (UniqueName: \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\") pod \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " Oct 30 22:04:14.625308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.621518 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\") pod \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " Oct 30 22:04:14.625308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.621602 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\") pod \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\" (UID: \"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5\") " Oct 30 22:04:14.654917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.654883 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca" pod "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" (UID: "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.764462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.764403 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca" pod "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" (UID: "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.769908 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" event={"ID":"ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5","Type":"ContainerDied","Data":"27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0"} Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.769998 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27606d9a6e52a72cd780d99c007e7ed07b52cdfd105a083a8ce94a98bbc993a0" Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.770101 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp" Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.772936 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.772965 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.802396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.773159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt" pod "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" (UID: "ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5"). InnerVolumeSpecName "kube-api-access-xzprt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.900446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:14.900402 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzprt\" (UniqueName: \"kubernetes.io/projected/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5-kube-api-access-xzprt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.539504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:15.538836 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.625647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:15.625577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.638715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:15.638660 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.965387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:15.965341 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.021432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.020661 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:04:16.332314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.323395 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 22:04:16.541328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.486551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.570721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.570678 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:04:16.611472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.611377 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.637315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.628415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:16.637315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.629023 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.676290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.658430 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.676290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.658520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.676290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.658937 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.812292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.796546 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.812292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.810589 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:16.812292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.810704 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:16.812292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.810750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:16.974729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.974691 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:16.989541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:16.982289 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:17.003848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.003821 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:17.175827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.175800 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:17.196290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.196247 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:17.275461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.275378 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:17.283501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.275685 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:17.293299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.293058 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:17.311710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.311676 5036 generic.go:356] "Generic (PLEG): container finished" podID="33e44091-8681-421c-906f-b145c25aa6a3" containerID="373783f7c42b3ce58452dc9e0262220fa76871b5957ba6a75680593aa643063c" exitCode=0 Oct 30 22:04:17.355291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.311938 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" event={"ID":"33e44091-8681-421c-906f-b145c25aa6a3","Type":"ContainerDied","Data":"373783f7c42b3ce58452dc9e0262220fa76871b5957ba6a75680593aa643063c"} Oct 30 22:04:17.355291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.349199 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm"] Oct 30 22:04:17.379139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.370983 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" containerName="revoke-keys" Oct 30 22:04:17.379139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.371016 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" containerName="revoke-keys" Oct 30 22:04:17.379139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.371193 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" containerName="revoke-keys" Oct 30 22:04:17.379846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.379824 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm"] Oct 30 22:04:17.380139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.380060 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.387826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.387806 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:17.388094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.388076 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:17.393460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.392530 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:04:17.393460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.392592 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:17.476722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.476694 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 22:04:17.484205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.484168 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6dm\" (UniqueName: \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.484415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.484396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.484557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.484537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.587149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587068 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587131 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587335 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587413 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587445 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.587626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587602 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587712 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.587806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\") pod \"33e44091-8681-421c-906f-b145c25aa6a3\" (UID: \"33e44091-8681-421c-906f-b145c25aa6a3\") " Oct 30 22:04:17.587903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.587886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.588082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588066 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.588155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.588310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588254 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.588527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6dm\" (UniqueName: \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.588527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588470 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:17.588739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588630 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.588739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588650 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e44091-8681-421c-906f-b145c25aa6a3-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.588739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588663 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.588739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588677 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.588739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.588692 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.589494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.589469 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.590139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.590113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.591886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.591862 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.593147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.593113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:17.593147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.593122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "kube-api-access-2nt7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:17.594720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.594697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token" pod "33e44091-8681-421c-906f-b145c25aa6a3" (UID: "33e44091-8681-421c-906f-b145c25aa6a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:17.600736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.600714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6dm\" (UniqueName: \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\") pod \"maas-api-revoke-sub-d4c4750b-h46nm\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.689915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.689877 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt7z\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-kube-api-access-2nt7z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.689915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.689911 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e44091-8681-421c-906f-b145c25aa6a3-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.690355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.689924 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e44091-8681-421c-906f-b145c25aa6a3-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.690355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.689940 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e44091-8681-421c-906f-b145c25aa6a3-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.707512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.707484 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:17.861541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.861452 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh"] Oct 30 22:04:17.862481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.862455 5036 cpu_manager.go:414] "containerMap: removing container" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" Oct 30 22:04:17.862481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.862476 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" Oct 30 22:04:17.862707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.862594 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e44091-8681-421c-906f-b145c25aa6a3" containerName="istio-proxy" Oct 30 22:04:17.867796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.867753 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:17.874747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.874718 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh"] Oct 30 22:04:17.994086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.994047 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:17.994299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.994113 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:17.994363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:17.994313 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgdw\" (UniqueName: \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.017094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.017072 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm"] Oct 30 22:04:18.095864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.095834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgdw\" (UniqueName: \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.096032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.096018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.096075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.096049 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.096686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.096664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.098634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.098611 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.100815 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:04:18.100783 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:04:18.107934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.107916 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgdw\" (UniqueName: \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\") pod \"maas-api-revoke-sub-351f7b5b-xtprh\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.187458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.187429 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:18.365092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.365059 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" event={"ID":"8609cb16-f71b-4202-9c52-7165cc1bc49c","Type":"ContainerStarted","Data":"63c202f509130df6d12b823b748af5f977806188b3a6892b4b163a18dc2906c9"} Oct 30 22:04:18.366758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.366738 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" Oct 30 22:04:18.367008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.366752 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr" event={"ID":"33e44091-8681-421c-906f-b145c25aa6a3","Type":"ContainerDied","Data":"8350c9eb27d2ee2d55c929e242f7464f41f832efa448c3ae45d0c1169db2ca3a"} Oct 30 22:04:18.367008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.366810 5036 scope.go:122] "RemoveContainer" containerID="373783f7c42b3ce58452dc9e0262220fa76871b5957ba6a75680593aa643063c" Oct 30 22:04:18.388559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.388535 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 22:04:18.394840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.394817 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-z6pqr"] Oct 30 22:04:18.491476 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:18.491438 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530a7381_cfac_4a3b_8047_ae21e777a5f2.slice/crio-e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72 WatchSource:0}: Error finding container e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72: Status 404 returned error can't find the container with id e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72 Oct 30 22:04:18.492665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:18.492637 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh"] Oct 30 22:04:19.377160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:19.377122 5036 generic.go:356] "Generic (PLEG): container finished" podID="8609cb16-f71b-4202-9c52-7165cc1bc49c" containerID="a7faf7b05b53218dcfc391aa63e42447d2190edaa71cebe539b2e811b305c92d" exitCode=0 Oct 30 22:04:19.377623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:19.377172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" event={"ID":"8609cb16-f71b-4202-9c52-7165cc1bc49c","Type":"ContainerDied","Data":"a7faf7b05b53218dcfc391aa63e42447d2190edaa71cebe539b2e811b305c92d"} Oct 30 22:04:19.378991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:19.378965 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" event={"ID":"530a7381-cfac-4a3b-8047-ae21e777a5f2","Type":"ContainerStarted","Data":"69b8e1a9991c75e1af281cd0c4f4f664e33618694a9a5ed2c1ac497bc951874d"} Oct 30 22:04:19.379117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:19.379010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" event={"ID":"530a7381-cfac-4a3b-8047-ae21e777a5f2","Type":"ContainerStarted","Data":"e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72"} Oct 30 22:04:19.411402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:19.411353 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" podStartSLOduration=2.411338561 podStartE2EDuration="2.411338561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:19.410957193 +0000 UTC m=+1483.360912222" watchObservedRunningTime="2026-10-30 22:04:19.411338561 +0000 UTC m=+1483.361293518" Oct 30 22:04:20.191338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.191305 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e44091-8681-421c-906f-b145c25aa6a3" path="/var/lib/kubelet/pods/33e44091-8681-421c-906f-b145c25aa6a3/volumes" Oct 30 22:04:20.391580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.391544 5036 generic.go:356] "Generic (PLEG): container finished" podID="530a7381-cfac-4a3b-8047-ae21e777a5f2" containerID="69b8e1a9991c75e1af281cd0c4f4f664e33618694a9a5ed2c1ac497bc951874d" exitCode=0 Oct 30 22:04:20.391965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.391624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" event={"ID":"530a7381-cfac-4a3b-8047-ae21e777a5f2","Type":"ContainerDied","Data":"69b8e1a9991c75e1af281cd0c4f4f664e33618694a9a5ed2c1ac497bc951874d"} Oct 30 22:04:20.581374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.581348 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:20.636758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.636713 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\" (UniqueName: \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\") pod \"8609cb16-f71b-4202-9c52-7165cc1bc49c\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " Oct 30 22:04:20.637053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.637020 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\") pod \"8609cb16-f71b-4202-9c52-7165cc1bc49c\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " Oct 30 22:04:20.637176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.637093 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\") pod \"8609cb16-f71b-4202-9c52-7165cc1bc49c\" (UID: \"8609cb16-f71b-4202-9c52-7165cc1bc49c\") " Oct 30 22:04:20.637764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.637710 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca" pod "8609cb16-f71b-4202-9c52-7165cc1bc49c" (UID: "8609cb16-f71b-4202-9c52-7165cc1bc49c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:20.639355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.639331 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca" pod "8609cb16-f71b-4202-9c52-7165cc1bc49c" (UID: "8609cb16-f71b-4202-9c52-7165cc1bc49c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:20.639730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.639701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm" pod "8609cb16-f71b-4202-9c52-7165cc1bc49c" (UID: "8609cb16-f71b-4202-9c52-7165cc1bc49c"). InnerVolumeSpecName "kube-api-access-pw6dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.739352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.739258 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.739352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.739301 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8609cb16-f71b-4202-9c52-7165cc1bc49c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.739352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.739312 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6dm\" (UniqueName: \"kubernetes.io/projected/8609cb16-f71b-4202-9c52-7165cc1bc49c-kube-api-access-pw6dm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.991360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.991282 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc"] Oct 30 22:04:20.992035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.992017 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8609cb16-f71b-4202-9c52-7165cc1bc49c" containerName="revoke-keys" Oct 30 22:04:20.992100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.992036 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8609cb16-f71b-4202-9c52-7165cc1bc49c" containerName="revoke-keys" Oct 30 22:04:20.992180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.992170 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8609cb16-f71b-4202-9c52-7165cc1bc49c" containerName="revoke-keys" Oct 30 22:04:20.996345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:20.996316 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.024457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.013914 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc"] Oct 30 22:04:21.165360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.165321 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.165531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.165368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4jt\" (UniqueName: \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.165572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.165545 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.266784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.266681 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.266947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.266928 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.266990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.266974 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4jt\" (UniqueName: \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.267660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.267635 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.269472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.269449 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.279720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.279694 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4jt\" (UniqueName: \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\") pod \"maas-api-revoke-sub-71f7b094-tt6lc\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.365542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.365510 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:21.403604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.403568 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" Oct 30 22:04:21.403945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.403609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm" event={"ID":"8609cb16-f71b-4202-9c52-7165cc1bc49c","Type":"ContainerDied","Data":"63c202f509130df6d12b823b748af5f977806188b3a6892b4b163a18dc2906c9"} Oct 30 22:04:21.403945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.403656 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c202f509130df6d12b823b748af5f977806188b3a6892b4b163a18dc2906c9" Oct 30 22:04:21.615500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.615473 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:21.670502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.670473 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc"] Oct 30 22:04:21.674132 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:21.674090 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0bd096_3797_43ba_a0b7_d16d64f310d3.slice/crio-d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0 WatchSource:0}: Error finding container d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0: Status 404 returned error can't find the container with id d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0 Oct 30 22:04:21.774344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.774255 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\" (UniqueName: \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\") pod \"530a7381-cfac-4a3b-8047-ae21e777a5f2\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " Oct 30 22:04:21.774530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.774500 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\") pod \"530a7381-cfac-4a3b-8047-ae21e777a5f2\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " Oct 30 22:04:21.774621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.774598 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\") pod \"530a7381-cfac-4a3b-8047-ae21e777a5f2\" (UID: \"530a7381-cfac-4a3b-8047-ae21e777a5f2\") " Oct 30 22:04:21.775052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.775024 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca" pod "530a7381-cfac-4a3b-8047-ae21e777a5f2" (UID: "530a7381-cfac-4a3b-8047-ae21e777a5f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:21.775394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.775374 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.777702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.777669 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw" pod "530a7381-cfac-4a3b-8047-ae21e777a5f2" (UID: "530a7381-cfac-4a3b-8047-ae21e777a5f2"). InnerVolumeSpecName "kube-api-access-pmgdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:21.778036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.777689 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca" pod "530a7381-cfac-4a3b-8047-ae21e777a5f2" (UID: "530a7381-cfac-4a3b-8047-ae21e777a5f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:21.876803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.876732 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmgdw\" (UniqueName: \"kubernetes.io/projected/530a7381-cfac-4a3b-8047-ae21e777a5f2-kube-api-access-pmgdw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.876803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:21.876759 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/530a7381-cfac-4a3b-8047-ae21e777a5f2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:22.454208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.454089 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" event={"ID":"fe0bd096-3797-43ba-a0b7-d16d64f310d3","Type":"ContainerStarted","Data":"ab21d505bf066934d35727aa39030afce23b3421c04995b5550038549c7778d9"} Oct 30 22:04:22.454208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.454177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" event={"ID":"fe0bd096-3797-43ba-a0b7-d16d64f310d3","Type":"ContainerStarted","Data":"d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0"} Oct 30 22:04:22.457895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.457837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" event={"ID":"530a7381-cfac-4a3b-8047-ae21e777a5f2","Type":"ContainerDied","Data":"e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72"} Oct 30 22:04:22.458082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.457910 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61ab6b36cca04eef21c320235c925f2ce78576aac92a7698d930f1b30695a72" Oct 30 22:04:22.458082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.458042 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh" Oct 30 22:04:22.479829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:22.479765 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" podStartSLOduration=2.479751867 podStartE2EDuration="2.479751867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:22.475471674 +0000 UTC m=+1486.425426621" watchObservedRunningTime="2026-10-30 22:04:22.479751867 +0000 UTC m=+1486.429706814" Oct 30 22:04:23.389348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.389313 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:04:23.390031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.390010 5036 cpu_manager.go:414] "containerMap: removing container" podUID="530a7381-cfac-4a3b-8047-ae21e777a5f2" containerName="revoke-keys" Oct 30 22:04:23.390161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.390033 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530a7381-cfac-4a3b-8047-ae21e777a5f2" containerName="revoke-keys" Oct 30 22:04:23.390257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.390242 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="530a7381-cfac-4a3b-8047-ae21e777a5f2" containerName="revoke-keys" Oct 30 22:04:23.393551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.393530 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.416901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.416875 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:04:23.470853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.470816 5036 generic.go:356] "Generic (PLEG): container finished" podID="fe0bd096-3797-43ba-a0b7-d16d64f310d3" containerID="ab21d505bf066934d35727aa39030afce23b3421c04995b5550038549c7778d9" exitCode=0 Oct 30 22:04:23.471225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.470859 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" event={"ID":"fe0bd096-3797-43ba-a0b7-d16d64f310d3","Type":"ContainerDied","Data":"ab21d505bf066934d35727aa39030afce23b3421c04995b5550038549c7778d9"} Oct 30 22:04:23.505838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.505810 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.505843 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506109 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506170 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506236 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4ch\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.506409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.506345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.607868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.607828 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4ch\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.607890 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.607940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.607991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608017 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608298 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608477 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608594 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608747 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.608967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.608853 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.610753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.610727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.611038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.611017 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.620721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.620695 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.620828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.620765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4ch\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:23.708383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:23.708303 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:24.019795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.019710 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:04:24.031307 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:24.031259 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b177cd1_0be4_49f1_b550_6d896570a85b.slice/crio-bdd7957bf828dcd5e53b7880a9d53c52557781fb5e2215ddb87586801c820b77 WatchSource:0}: Error finding container bdd7957bf828dcd5e53b7880a9d53c52557781fb5e2215ddb87586801c820b77: Status 404 returned error can't find the container with id bdd7957bf828dcd5e53b7880a9d53c52557781fb5e2215ddb87586801c820b77 Oct 30 22:04:24.034384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.034341 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:24.034502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.034429 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:24.034502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.034469 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:24.486562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.486514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" event={"ID":"5b177cd1-0be4-49f1-b550-6d896570a85b","Type":"ContainerStarted","Data":"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3"} Oct 30 22:04:24.487235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.487210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" event={"ID":"5b177cd1-0be4-49f1-b550-6d896570a85b","Type":"ContainerStarted","Data":"bdd7957bf828dcd5e53b7880a9d53c52557781fb5e2215ddb87586801c820b77"} Oct 30 22:04:24.552304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.551922 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podStartSLOduration=1.551902588 podStartE2EDuration="1.551902588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:24.540776106 +0000 UTC m=+1488.490731085" watchObservedRunningTime="2026-10-30 22:04:24.551902588 +0000 UTC m=+1488.501857535" Oct 30 22:04:24.713150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.713114 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:24.789831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.789757 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:24.871975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.871942 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\" (UniqueName: \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\") pod \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " Oct 30 22:04:24.872162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.872001 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\") pod \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " Oct 30 22:04:24.872162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.872034 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\") pod \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\" (UID: \"fe0bd096-3797-43ba-a0b7-d16d64f310d3\") " Oct 30 22:04:24.872488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.872458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca" pod "fe0bd096-3797-43ba-a0b7-d16d64f310d3" (UID: "fe0bd096-3797-43ba-a0b7-d16d64f310d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:24.872785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.872759 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:24.874676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.874650 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt" pod "fe0bd096-3797-43ba-a0b7-d16d64f310d3" (UID: "fe0bd096-3797-43ba-a0b7-d16d64f310d3"). InnerVolumeSpecName "kube-api-access-sb4jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:24.874861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.874831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca" pod "fe0bd096-3797-43ba-a0b7-d16d64f310d3" (UID: "fe0bd096-3797-43ba-a0b7-d16d64f310d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:24.973617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.973581 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb4jt\" (UniqueName: \"kubernetes.io/projected/fe0bd096-3797-43ba-a0b7-d16d64f310d3-kube-api-access-sb4jt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:24.973617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:24.973612 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0bd096-3797-43ba-a0b7-d16d64f310d3-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.056320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.056194 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:04:25.057570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.057532 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fe0bd096-3797-43ba-a0b7-d16d64f310d3" containerName="revoke-keys" Oct 30 22:04:25.057735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.057721 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0bd096-3797-43ba-a0b7-d16d64f310d3" containerName="revoke-keys" Oct 30 22:04:25.058071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.058048 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0bd096-3797-43ba-a0b7-d16d64f310d3" containerName="revoke-keys" Oct 30 22:04:25.086525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.086492 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:04:25.086659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.086610 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.096355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.096150 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-18ba2f-openshift-default-dockercfg-6d5d5\"" Oct 30 22:04:25.176835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.176796 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf9t\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.176844 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177030 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177242 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.177550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.177538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf9t\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290421 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290507 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290539 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290687 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290722 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.290866 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.291385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.292613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.291781 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.294660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.294638 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.294837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.294690 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.308332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.303985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.340292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.338840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.364293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.341128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf9t\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.364293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.345894 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.364293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.353615 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.407293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.406523 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:25.510076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.507001 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" Oct 30 22:04:25.510076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.507233 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc" event={"ID":"fe0bd096-3797-43ba-a0b7-d16d64f310d3","Type":"ContainerDied","Data":"d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0"} Oct 30 22:04:25.510076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.507396 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9624571e559a5a815b84db6c7cc0f5e2ab6e087daa1d8f9a7368775e46509f0" Oct 30 22:04:25.756084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.755257 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:25.756084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.755336 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:25.763600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.763574 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:04:25.769677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.769653 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.788103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.788075 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:04:25.808764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.808726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5qq\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.808904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.808819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.808904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.808864 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.808973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.808909 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.809010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.808998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.809047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.809029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.809176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.809159 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.809493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.809470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.809552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.809513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.857156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.856773 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:25.857156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.856839 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:25.857156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.856869 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:25.858923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.858901 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916202 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916444 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5qq\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.916899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.916659 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.917099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.917075 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.922232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.917316 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.922232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.917782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.922232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.918787 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.922232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.919039 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.922232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.922170 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.923612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.923562 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.947594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.947540 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:25.947930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:25.947909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5qq\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:26.103300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.103185 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:26.507481 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:26.507433 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0abe737_9a76_4ee3_b0c0_166e4297b30b.slice/crio-254a9e1b7713c61dff1bb867c6a2e6761ac06ca52be4c2a5796b9184596656b6 WatchSource:0}: Error finding container 254a9e1b7713c61dff1bb867c6a2e6761ac06ca52be4c2a5796b9184596656b6: Status 404 returned error can't find the container with id 254a9e1b7713c61dff1bb867c6a2e6761ac06ca52be4c2a5796b9184596656b6 Oct 30 22:04:26.507869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.507841 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:04:26.533696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.511581 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:26.533696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.511670 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:26.533696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.511748 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:26.561834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.561723 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerStarted","Data":"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b"} Oct 30 22:04:26.561834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.561812 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerStarted","Data":"2ae43212ae5ea2c18d7eda948680861707fe1588aec53bc2acf471e2447a97f7"} Oct 30 22:04:26.589829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.589776 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podStartSLOduration=1.589763096 podStartE2EDuration="1.589763096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:26.583732386 +0000 UTC m=+1490.533687342" watchObservedRunningTime="2026-10-30 22:04:26.589763096 +0000 UTC m=+1490.539718041" Oct 30 22:04:26.716290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.711762 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:26.716290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:26.711825 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:27.428613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.428576 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:27.448647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.448612 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:27.698670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.698585 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" event={"ID":"b0abe737-9a76-4ee3-b0c0-166e4297b30b","Type":"ContainerStarted","Data":"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4"} Oct 30 22:04:27.699597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.699574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" event={"ID":"b0abe737-9a76-4ee3-b0c0-166e4297b30b","Type":"ContainerStarted","Data":"254a9e1b7713c61dff1bb867c6a2e6761ac06ca52be4c2a5796b9184596656b6"} Oct 30 22:04:27.699809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.699794 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:27.700464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.700448 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:04:27.708745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.708719 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:27.708847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.708792 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:27.797883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:27.795989 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podStartSLOduration=2.795975469 podStartE2EDuration="2.795975469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:27.771089474 +0000 UTC m=+1491.721044422" watchObservedRunningTime="2026-10-30 22:04:27.795975469 +0000 UTC m=+1491.745930415" Oct 30 22:04:28.103631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:28.103506 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:28.105057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:28.105036 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:28.105124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:28.105085 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:28.709160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:28.709128 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:28.709776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:28.709738 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:29.105352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.104393 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:29.105352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.104483 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:29.709583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.709549 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:29.709986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.709608 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:29.809918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.809887 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:04:29.888883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.888853 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:04:29.889367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.889290 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:29.895951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:29.895928 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cca1c7-openshift-default-dockercfg-lmtw8\"" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.106730 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.106793 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144603 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144682 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144821 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144862 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144904 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tr4\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.144946 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.145131 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.243061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.145161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452655 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452709 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452819 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452858 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.452978 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.453022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tr4\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.453290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.453062 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.455063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.453970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.455665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.455645 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.455799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.455773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.456042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.456019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.456445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.456424 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.458257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.458239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.458833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.458810 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.466522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.466501 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.468153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.468128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tr4\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.585211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.585129 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:30.709715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.709177 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 22:04:30.709715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.709239 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 22:04:30.846166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.845786 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:04:30.873957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.864152 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:30.877955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.877856 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:30.878337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.878256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vbm\" (UniqueName: \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:30.879902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.879870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:04:30.880631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.880475 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-18ba2f-dockercfg-hgtxn\"" Oct 30 22:04:30.950290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.949605 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-18ba2f\"" Oct 30 22:04:31.002290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.987457 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vbm\" (UniqueName: \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:31.002290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:30.990922 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:31.022326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.019632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:31.093448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.090031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vbm\" (UniqueName: \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\") pod \"payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:31.093448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.091653 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:31.093448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.091749 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:31.093448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.091795 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:31.188042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.188013 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:31.188257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.188231 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:31.199619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.197206 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:31.227290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.216647 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:04:31.698121 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:31.697327 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda050b519_eca1_4791_b519_84014afcb4bb.slice/crio-b6f3e764d808311479685768ab2f6f8cfe0bc6b9289c1ad603317de857ea2299 WatchSource:0}: Error finding container b6f3e764d808311479685768ab2f6f8cfe0bc6b9289c1ad603317de857ea2299: Status 404 returned error can't find the container with id b6f3e764d808311479685768ab2f6f8cfe0bc6b9289c1ad603317de857ea2299 Oct 30 22:04:31.700683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.700656 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:04:31.831221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.829968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerStarted","Data":"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e"} Oct 30 22:04:31.831221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.830031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerStarted","Data":"d670128767d09566e80eec84ac0b58d7b65ac0e3fa58c0446a32de23555f4e69"} Oct 30 22:04:31.850613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.850533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" event={"ID":"a050b519-eca1-4791-b519-84014afcb4bb","Type":"ContainerStarted","Data":"b6f3e764d808311479685768ab2f6f8cfe0bc6b9289c1ad603317de857ea2299"} Oct 30 22:04:31.882366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:31.881704 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podStartSLOduration=2.881681907 podStartE2EDuration="2.881681907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:31.873762571 +0000 UTC m=+1495.823717537" watchObservedRunningTime="2026-10-30 22:04:31.881681907 +0000 UTC m=+1495.831636855" Oct 30 22:04:32.106726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.106627 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:32.106726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.106688 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:32.595294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.591342 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:32.716399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.716357 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:32.716614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.716431 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:32.716769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.716747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:32.876792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.876709 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:04:32.884129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.884100 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" event={"ID":"a050b519-eca1-4791-b519-84014afcb4bb","Type":"ContainerStarted","Data":"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c"} Oct 30 22:04:32.884401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.884380 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:32.884676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.884660 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:32.884856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.884841 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:04:32.884971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.884610 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.892507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.892468 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5313f0-openshift-default-dockercfg-scpj8\"" Oct 30 22:04:32.903066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.903037 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.933826 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" podStartSLOduration=2.933804203 podStartE2EDuration="2.933804203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:32.906661286 +0000 UTC m=+1496.856616233" watchObservedRunningTime="2026-10-30 22:04:32.933804203 +0000 UTC m=+1496.883759150" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959296 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959442 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959517 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959682 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.959811 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.960608 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw57\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:32.962392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:32.960663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067334 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw57\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067452 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067504 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.068327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.067628 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.073426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.073395 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.073983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.073955 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.089292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.077185 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.089292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.077465 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.089292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.079377 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.089292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.080101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.104038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.103994 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:33.104238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.104062 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:33.200396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.156045 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.222292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.211920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.383625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.383585 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw57\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.385817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.385785 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:33.386217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.221075 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:33.389534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.389514 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:33.392745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.392065 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:04:33.733111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.733080 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:04:33.733620 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:04:33.733591 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66837898_383c_4ac7_93bb_29b5de1a190b.slice/crio-e56b87479507e6a280066f0d5a16b1e660b2be4fc1f60969979708482e0ddbdd WatchSource:0}: Error finding container e56b87479507e6a280066f0d5a16b1e660b2be4fc1f60969979708482e0ddbdd: Status 404 returned error can't find the container with id e56b87479507e6a280066f0d5a16b1e660b2be4fc1f60969979708482e0ddbdd Oct 30 22:04:33.741276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.741221 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:33.741360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.741337 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:33.741414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.741389 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:33.895969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.895930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerStarted","Data":"0216a64334c9859397c9b98badef74818f1d074822cd12d113fb582cde86f208"} Oct 30 22:04:33.896476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.895999 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerStarted","Data":"e56b87479507e6a280066f0d5a16b1e660b2be4fc1f60969979708482e0ddbdd"} Oct 30 22:04:33.935372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:33.935311 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podStartSLOduration=1.935290087 podStartE2EDuration="1.935290087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:33.918754768 +0000 UTC m=+1497.868709744" watchObservedRunningTime="2026-10-30 22:04:33.935290087 +0000 UTC m=+1497.885245034" Oct 30 22:04:34.104886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.104796 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:04:34.105161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.105127 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 22:04:34.390778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.390752 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:34.396234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.396209 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:34.908002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.907963 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:34.910173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:34.910145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:04:35.173381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.171617 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:35.173381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.172159 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:35.173381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.173196 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:04:35.835563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.835497 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:04:35.841704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.841683 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:35.862964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.862930 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cca1c7-dockercfg-6bdpz\"" Oct 30 22:04:35.866367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.863465 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:04:35.910403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.910368 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cca1c7\"" Oct 30 22:04:35.957708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.957674 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkx2k\" (UniqueName: \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:35.958163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:35.958139 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.076746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.076702 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.078176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.078149 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkx2k\" (UniqueName: \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.078877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.077911 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.154141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.154110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkx2k\" (UniqueName: \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\") pod \"payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.195291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.183798 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cca1c7-dockercfg-6bdpz\"" Oct 30 22:04:36.201296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.195801 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.614955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.612331 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:04:36.750414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.740633 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:04:36.762435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.762409 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:04:36.931140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.931106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" event={"ID":"79f41ba0-9ecf-4a97-aa9c-7515e322cb72","Type":"ContainerStarted","Data":"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59"} Oct 30 22:04:36.931693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.931663 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" event={"ID":"79f41ba0-9ecf-4a97-aa9c-7515e322cb72","Type":"ContainerStarted","Data":"0133d83aee7ac56b210cf861864e916fc783d8d3cdd7bdc99823579ac2d69d7e"} Oct 30 22:04:36.931790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.931709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:36.974643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:36.973649 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" podStartSLOduration=1.973625528 podStartE2EDuration="1.973625528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:36.958700627 +0000 UTC m=+1500.908655576" watchObservedRunningTime="2026-10-30 22:04:36.973625528 +0000 UTC m=+1500.923580492" Oct 30 22:04:38.321294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.313393 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:04:38.708388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.663523 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:38.816288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.714154 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5313f0-dockercfg-w5j98\"" Oct 30 22:04:38.816288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.736347 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5313f0\"" Oct 30 22:04:38.936302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.914239 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:38.957305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:38.936648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkh78\" (UniqueName: \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.060564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.060527 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkh78\" (UniqueName: \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.061249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.061222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.063064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.063043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.319749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.319667 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkh78\" (UniqueName: \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\") pod \"payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.340600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.324710 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.362298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.349367 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:04:39.367290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.358741 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:39.520548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.520502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.521362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.521334 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:04:39.522814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.522788 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.918686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.918573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.953291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.942359 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.990423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:39.990372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.130896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.130843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.240351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.239674 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.321492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.288970 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 30 22:04:40.335308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.328339 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.528644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.526532 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.528644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.526955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:40.711079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.711039 5036 scope.go:122] "RemoveContainer" containerID="da70cc011745add90315264844bf7634d1a64cee16c23dd1a5d77aeb42c6d20d" Oct 30 22:04:40.763004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.762958 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.763572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.763546 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:40.763760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.763747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:04:40.763893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.763881 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:40.922907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.922604 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.979182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.979141 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.981158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.979253 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:04:40.981158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.980021 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:41.001325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:40.993457 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:41.003425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.002281 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:41.353420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.353381 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:41.570375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.566389 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:41.570375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.566721 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:04:41.985303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.982898 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:41.985590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:41.985301 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:04:42.028140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.017673 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:42.055549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.053988 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:42.493873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.490776 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:42.569898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.569852 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:42.580160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.572042 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:42.726759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.691551 5036 scope.go:122] "RemoveContainer" containerID="167144291d3229c9b95c273aae27316b3d34032ed5aca1d767ab16089e7dea77" Oct 30 22:04:43.060683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:42.999386 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:43.060683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.035195 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:43.062003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.061929 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:04:43.078320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.078239 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:43.079204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.079185 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:04:43.266696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.266169 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:04:43.321375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.320576 5036 scope.go:122] "RemoveContainer" containerID="7dc53e76b436167f66f3ef19f0fde23e6cd225621f3d014bdddee9f5d2b690cc" Oct 30 22:04:43.429701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.429653 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:04:43.530716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.520684 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:43.652928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.651579 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:43.661393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.661227 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:04:43.740776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.728053 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:04:43.786292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.771883 5036 scope.go:122] "RemoveContainer" containerID="f097631355c180a422c315eeb0d7e26b83771a2bb276ea59afea90056e50ab73" Oct 30 22:04:43.833194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.833164 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:43.993087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.991760 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:04:43.995870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.995706 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:04:43.997335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:43.997302 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:04:44.001526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.001114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvwg\" (UniqueName: \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.001526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.001280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.004082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.004064 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ecfbd6\"" Oct 30 22:04:44.008018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.008002 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:04:44.008257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.008242 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:04:44.008482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.008467 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:04:44.027440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.014965 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ecfbd6-dockercfg-rwn22\"" Oct 30 22:04:44.027440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.023677 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:04:44.044581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.044536 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:04:44.046529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.046511 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.060319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.059997 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:04:44.117284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.117223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.117597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.117569 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvwg\" (UniqueName: \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.117888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.117865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.141398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.121511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.148119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.121873 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc67l\" (UniqueName: \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.148119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.129877 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" event={"ID":"6c3fc258-f621-46c3-8d79-efe924508ee5","Type":"ContainerStarted","Data":"7af506e86ef79d6a6a596f95c020ae25d316a4af8cc1a208389e8c2b2d488390"} Oct 30 22:04:44.194064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.189634 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvwg\" (UniqueName: \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\") pod \"payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.250313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.250257 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc67l\" (UniqueName: \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.250485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.250463 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.253689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.253661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.267526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.267492 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc67l\" (UniqueName: \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\") pod \"payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.428795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.428756 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:44.520373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.515715 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:44.748919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.746498 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:04:44.876331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.867626 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:04:44.920330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:44.917608 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:04:45.143554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.143506 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" event={"ID":"6c3fc258-f621-46c3-8d79-efe924508ee5","Type":"ContainerStarted","Data":"eb99206f09bd7f327938fb280d898de79d7d3121fd64e2b9480b327d2df55090"} Oct 30 22:04:45.143766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.143575 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:45.145606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.145575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" event={"ID":"ef5799f4-8428-4ea4-98bc-570c933a9b5a","Type":"ContainerStarted","Data":"5a4ebed3ba61c3846effb583552a2088d4ff45bf310f57c86a4a9490bd480018"} Oct 30 22:04:45.145776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.145624 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:04:45.145776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.145649 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" event={"ID":"ef5799f4-8428-4ea4-98bc-570c933a9b5a","Type":"ContainerStarted","Data":"62fe74992c2a6ccf7f1b349e89714ac47e39aaa4ad68932866fdf40cdd2ea205"} Oct 30 22:04:45.147635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.147597 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" event={"ID":"8519f1f5-1c5e-44e7-ad21-9f423750f1a7","Type":"ContainerStarted","Data":"83c53512533b378cba3f2d8ead713de4fce173bb436012c82a3dfe7de9e7b934"} Oct 30 22:04:45.147768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.147648 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:45.147768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.147666 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" event={"ID":"8519f1f5-1c5e-44e7-ad21-9f423750f1a7","Type":"ContainerStarted","Data":"376cb2f39e69feedd39e1b99028ac811046e4d8e5b6880c85f204415a493ea20"} Oct 30 22:04:45.169670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.169610 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" podStartSLOduration=8.169592052 podStartE2EDuration="8.169592052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:45.166164344 +0000 UTC m=+1509.116119290" watchObservedRunningTime="2026-10-30 22:04:45.169592052 +0000 UTC m=+1509.119546998" Oct 30 22:04:45.193138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.193083 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" podStartSLOduration=4.193069092 podStartE2EDuration="4.193069092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:45.192966207 +0000 UTC m=+1509.142921153" watchObservedRunningTime="2026-10-30 22:04:45.193069092 +0000 UTC m=+1509.143024037" Oct 30 22:04:45.216109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:45.216058 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" podStartSLOduration=4.216044123 podStartE2EDuration="4.216044123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:45.213922399 +0000 UTC m=+1509.163877371" watchObservedRunningTime="2026-10-30 22:04:45.216044123 +0000 UTC m=+1509.165999068" Oct 30 22:04:48.101088 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:04:48.101043 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:04:49.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:49.304448 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:04:56.162457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:56.162422 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:04:56.162970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:56.162515 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:04:56.162970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:04:56.162558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:05:00.392329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.390006 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:05:00.410292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.398009 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.435290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.435225 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0f73c9-openshift-default-dockercfg-n4lfd\"" Oct 30 22:05:00.444195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.442692 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:05:00.534566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.534522 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.534820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.534685 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.534820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.534743 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.534955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.534926 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.535008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.534997 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.535098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.535079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.535196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.535178 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.535373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.535353 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.535473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.535455 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrt9\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638215 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrt9\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638308 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638385 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638421 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638601 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.638643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.639012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638712 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.639012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.638804 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.642283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.639292 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.642283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.639986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.661368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.648879 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.661368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.649124 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.661368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.649721 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.686784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.686749 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.692300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.690007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrt9\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.692300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.690548 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.714293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.712304 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\") pod \"e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:00.776858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:00.776555 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:01.256291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.248551 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:05:01.307451 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:01.307398 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cd4692_c724_4687_b9cc_066b9c2c04d5.slice/crio-e3b8ab92627ebac496d1d34c763c683d22fa7cf718a14772ad9584ab9c36c4ea WatchSource:0}: Error finding container e3b8ab92627ebac496d1d34c763c683d22fa7cf718a14772ad9584ab9c36c4ea: Status 404 returned error can't find the container with id e3b8ab92627ebac496d1d34c763c683d22fa7cf718a14772ad9584ab9c36c4ea Oct 30 22:05:01.317652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.317606 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:01.317893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.317866 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:01.318051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.318033 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:01.498425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.498348 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerStarted","Data":"e3b8ab92627ebac496d1d34c763c683d22fa7cf718a14772ad9584ab9c36c4ea"} Oct 30 22:05:01.904290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.849390 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:05:01.904290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.879406 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:01.946307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.905522 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-38d6e2-dockercfg-j9mg4\"" Oct 30 22:05:01.946307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.920717 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-38d6e2\"" Oct 30 22:05:01.963299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.962973 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rg9q\" (UniqueName: \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:01.963299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:01.963163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.025786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.025754 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:05:02.089300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.084946 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rg9q\" (UniqueName: \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.089300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.085122 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.118235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.118202 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rg9q\" (UniqueName: \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.126903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.126872 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.322635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.322538 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:02.633451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.633400 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:02.680332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.680290 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerStarted","Data":"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7"} Oct 30 22:05:02.781417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.781386 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:03.021294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:02.999745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.021742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.021701 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.022171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.013347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.065289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.049419 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.095824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.095783 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:05:03.137301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.136854 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.201294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.176646 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:05:03.201294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.176838 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9c8ecf-openshift-default-dockercfg-6qtv6\"" Oct 30 22:05:03.391348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.362659 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.391921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.391887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.392125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.392108 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6m4\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.404598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.401326 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.414348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.405587 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.414921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.414891 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.415143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.415120 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.415279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.415244 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.422520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.422480 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.422766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.422747 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.528637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.528589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.528956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.528936 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.529105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.529087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6m4\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.571292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.529244 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podStartSLOduration=3.529227793 podStartE2EDuration="3.529227793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:02.741126777 +0000 UTC m=+1526.691081724" watchObservedRunningTime="2026-10-30 22:05:03.529227793 +0000 UTC m=+1527.479182739" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.529324 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.591407 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.591510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.591541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.591639 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.591689 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.592164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.547933 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.548632 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.592830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.555020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.612291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.598169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.652538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.626242 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.652538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.633973 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6m4\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.652538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.650577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.652538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.651156 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:03.682541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.680352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.727852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.718609 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:03.813527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.792851 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:03.813527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.792911 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:03.817687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:03.817106 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.037167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.037542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.037823 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.038475 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.038537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.038579 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.054348 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.085395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.054783 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:05:04.304379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.284478 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:04.304379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.284550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.433305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.414971 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.687381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.671632 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:04.687381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.671738 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.717729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:04.717683 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:05.052413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.052296 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:05.076291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.069957 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:05.134425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.134382 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:05.364155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.364045 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:05:05.418289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.394340 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:05.603294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.481365 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:05.949346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:05.867629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded" Oct 30 22:05:06.072383 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:06.072337 5036 logging.go:55] [core] [Channel #3767 SubChannel #3768]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: write tcp 10.132.0.2:40538->10.132.0.70:50052: use of closed network connection Oct 30 22:05:06.072751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.072715 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:06.132959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.109360 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:06.169346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.159158 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:05:06.256985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.256885 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-9qcwk" podUID="e388b1ae-8405-44ab-949b-4d0d12da7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 22:05:06.270458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.270414 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" podUID="0b9abb0b-7989-467e-86a9-41bbe7ce3a59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 22:05:06.271094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.271058 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:06.436293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.425837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:06.557346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.556555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w9r5n" podUID="0768611d-547a-42bd-b85b-65ae92070f3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:06.616350 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:05:06.585138 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.298s" Oct 30 22:05:06.803570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.803538 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:05:06.804096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.804065 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" event={"ID":"d65e758b-0bc0-46f3-87e6-751b7108599c","Type":"ContainerStarted","Data":"67df52100810a387a73561b63d7a5e4718cfb439e34e95c978eaba41d5a91e55"} Oct 30 22:05:06.804313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.804295 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:06.843352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.601819 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:06.843352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:06.601784 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:07.082300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.080301 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:07.082300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.080627 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded" Oct 30 22:05:07.117183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.115667 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:05:07.173296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.173153 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:07.296096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.295620 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:07.404479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.404440 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 22:05:07.408295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.404446 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:07.443438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.439349 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:07.545304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.522868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:05:07.653502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.653415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:07.713151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.713122 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:07.885399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.885367 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:05:07.885683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.859469 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:07.886118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:07.859405 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:08.106062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.089353 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:08.196351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.175734 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:05:08.196351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.176105 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:05:08.296320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.295923 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:08.447353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.445004 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:08.573573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.573541 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:05:08.575725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.575703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:08.640330 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:08.627220 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe6040c_17a5_4962_aa05_e41be16529c5.slice/crio-64d3de649f1f21e2d3137dc6f89376a0011716f787c95932511b199fdbf3da22 WatchSource:0}: Error finding container 64d3de649f1f21e2d3137dc6f89376a0011716f787c95932511b199fdbf3da22: Status 404 returned error can't find the container with id 64d3de649f1f21e2d3137dc6f89376a0011716f787c95932511b199fdbf3da22 Oct 30 22:05:08.735346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.654229 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:08.735346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.654438 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:08.735346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.654518 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:08.999590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:08.999489 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:09.000985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.000949 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:09.245217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.245173 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:09.273825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.273704 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:05:09.278888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.278858 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:05:09.309294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.308531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:09.394292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.394223 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 22:05:09.478292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.472638 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:09.680779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.556698 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:05:09.736964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.734311 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0f73c9-dockercfg-vl52z\"" Oct 30 22:05:09.768285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.763816 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0f73c9\"" Oct 30 22:05:09.799784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.799742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tz6\" (UniqueName: \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:09.807296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.801703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:09.862436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.862392 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:09.909947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.909863 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tz6\" (UniqueName: \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:09.924288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.910367 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:09.950294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:09.918230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:10.016298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.011550 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tz6\" (UniqueName: \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\") pod \"payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:10.053296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.025384 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:10.076682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.074084 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:10.141314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.103095 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:10.141314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.117637 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:10.167349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.154521 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:10.308530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.308482 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:05:10.335290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.312745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:10.336183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.309487 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:10.445402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.445303 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:05:10.471473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:10.469295 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:05:11.010959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.010920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerStarted","Data":"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7"} Oct 30 22:05:11.011308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.011249 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerStarted","Data":"64d3de649f1f21e2d3137dc6f89376a0011716f787c95932511b199fdbf3da22"} Oct 30 22:05:11.062290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.051362 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" event={"ID":"d65e758b-0bc0-46f3-87e6-751b7108599c","Type":"ContainerStarted","Data":"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf"} Oct 30 22:05:11.094501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.094473 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:11.094780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.091469 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:11.114300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.093461 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:05:11.114300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.065893 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podStartSLOduration=9.065872259 podStartE2EDuration="9.065872259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:11.063527517 +0000 UTC m=+1535.013482494" watchObservedRunningTime="2026-10-30 22:05:11.065872259 +0000 UTC m=+1535.015827206" Oct 30 22:05:11.114300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.104257 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:11.152308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.145041 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:05:11.342292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.341779 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:11.342292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.342187 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:11.400228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.400178 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:11.672292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.345253 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" podStartSLOduration=10.345235187 podStartE2EDuration="10.345235187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:11.153123477 +0000 UTC m=+1535.103078420" watchObservedRunningTime="2026-10-30 22:05:11.345235187 +0000 UTC m=+1535.295190134" Oct 30 22:05:11.848996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.848538 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:11.848996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.848673 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:05:11.848996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:11.848718 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:05:12.095774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.095643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:12.151871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.122372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:12.168947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.165742 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:12.352122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.352033 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:05:12.357399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.357365 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:05:12.432310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.402413 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:12.501291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.497359 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:05:12.850702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.850640 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:12.913848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.913759 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:05:12.943318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.942367 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" event={"ID":"1c463d01-e96e-43fe-a278-beb0b28b2c88","Type":"ContainerStarted","Data":"366b909bddae61719b7b7adeb4508e4ab409734d2fc0e4774a470dbaecd94391"} Oct 30 22:05:12.943318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:12.942447 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:13.149419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.149362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:13.171295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.167333 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:13.218785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.193940 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:05:13.233292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.219561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:13.433491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.433459 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:05:13.711412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.711371 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:13.737301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.711930 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:05:13.737843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.737806 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:13.738900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:13.738874 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:14.085241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.085143 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:05:14.174893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.174847 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:14.249196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.216367 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:14.283540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.247937 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:14.318464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.318430 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.363465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.356559 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-0a2577-dockercfg-jsbdj\"" Oct 30 22:05:14.382958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.382927 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0a2577\"" Oct 30 22:05:14.562291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.552146 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.563568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.563537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfmp\" (UniqueName: \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.676902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.676866 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:14.677085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.676927 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" event={"ID":"1c463d01-e96e-43fe-a278-beb0b28b2c88","Type":"ContainerStarted","Data":"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821"} Oct 30 22:05:14.677085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.676959 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:05:14.683536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.683292 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:05:14.684799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.684765 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.685016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.684999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfmp\" (UniqueName: \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.686315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.686287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.688637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.688612 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:05:14.700434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.700394 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:05:14.702837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.702809 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:05:14.709012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.708989 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:05:14.718290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.715347 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:14.731917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.729010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfmp\" (UniqueName: \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\") pod \"payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:14.742010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.741972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:14.769637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.768655 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:05:14.796992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.796935 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" podStartSLOduration=7.796916593 podStartE2EDuration="7.796916593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:14.682793126 +0000 UTC m=+1538.632748074" watchObservedRunningTime="2026-10-30 22:05:14.796916593 +0000 UTC m=+1538.746871540" Oct 30 22:05:14.980866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:14.980832 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:15.179421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.167208 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:05:15.187114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.187090 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.189175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.189143 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:05:15.195494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.195295 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9c8ecf\"" Oct 30 22:05:15.195494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.195459 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9c8ecf-dockercfg-r4fqc\"" Oct 30 22:05:15.344318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.321658 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.344318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.321914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9wh\" (UniqueName: \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.408289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.360482 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:05:15.408289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.361565 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.505336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.502742 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9wh\" (UniqueName: \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.505336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.502932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.505336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.503914 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.507677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.506581 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:05:15.528249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.528220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9wh\" (UniqueName: \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\") pod \"payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.528589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.528534 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:05:15.635328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.634354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" event={"ID":"48e2bcd2-1cd7-4429-bef9-054536865311","Type":"ContainerStarted","Data":"ef762eefe35348412115f1f0f35506b9f24d8f8d68611d7733274f85805877a3"} Oct 30 22:05:15.708421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.708068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.709006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.708969 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdw2\" (UniqueName: \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.804623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.804590 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:15.815462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.813795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.815462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.814109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdw2\" (UniqueName: \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.815462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.815088 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.836108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.836056 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdw2\" (UniqueName: \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\") pod \"payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:15.869945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:15.869672 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:16.139769 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:16.139719 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacedbaf4_45ce_4ca9_89f9_371f6bcfc44d.slice/crio-98fa59e13dee22000fced512270d3f36864ac4c4a6efbe71e5843622bdba08d4 WatchSource:0}: Error finding container 98fa59e13dee22000fced512270d3f36864ac4c4a6efbe71e5843622bdba08d4: Status 404 returned error can't find the container with id 98fa59e13dee22000fced512270d3f36864ac4c4a6efbe71e5843622bdba08d4 Oct 30 22:05:16.145737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.145711 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:05:16.216788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.216759 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:05:16.233480 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:16.232889 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e10f6d_a52a_41d0_9d15_41c01a8f3a93.slice/crio-97c9e461381eb1fd5b93804d3638e3fa9ee7c9e3acfa317c488c37a56bf61bc4 WatchSource:0}: Error finding container 97c9e461381eb1fd5b93804d3638e3fa9ee7c9e3acfa317c488c37a56bf61bc4: Status 404 returned error can't find the container with id 97c9e461381eb1fd5b93804d3638e3fa9ee7c9e3acfa317c488c37a56bf61bc4 Oct 30 22:05:16.654211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.654165 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" event={"ID":"48e2bcd2-1cd7-4429-bef9-054536865311","Type":"ContainerStarted","Data":"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8"} Oct 30 22:05:16.655342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.655318 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:16.657968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.657942 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" event={"ID":"41e10f6d-a52a-41d0-9d15-41c01a8f3a93","Type":"ContainerStarted","Data":"cca817d13b3349540d9fe5d2c236d762aa8aef1fa9535e9db56a7eaafc21e5a9"} Oct 30 22:05:16.658237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.658219 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" event={"ID":"41e10f6d-a52a-41d0-9d15-41c01a8f3a93","Type":"ContainerStarted","Data":"97c9e461381eb1fd5b93804d3638e3fa9ee7c9e3acfa317c488c37a56bf61bc4"} Oct 30 22:05:16.658526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.658489 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:16.660986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.660944 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" event={"ID":"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d","Type":"ContainerStarted","Data":"39f6dc0b97f7eb99687502baf0cefe466f74395603e679966732d2f54a2fdcb0"} Oct 30 22:05:16.661075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.660991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" event={"ID":"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d","Type":"ContainerStarted","Data":"98fa59e13dee22000fced512270d3f36864ac4c4a6efbe71e5843622bdba08d4"} Oct 30 22:05:16.661256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.661226 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:16.695139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.692692 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" podStartSLOduration=4.6926761710000005 podStartE2EDuration="4.692676171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.683425614 +0000 UTC m=+1540.633380561" watchObservedRunningTime="2026-10-30 22:05:16.692676171 +0000 UTC m=+1540.642631120" Oct 30 22:05:16.749784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.749689 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" podStartSLOduration=1.749671031 podStartE2EDuration="1.749671031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.721629657 +0000 UTC m=+1540.671584605" watchObservedRunningTime="2026-10-30 22:05:16.749671031 +0000 UTC m=+1540.699625977" Oct 30 22:05:16.759157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:16.759100 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" podStartSLOduration=1.759078862 podStartE2EDuration="1.759078862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.746946298 +0000 UTC m=+1540.696901244" watchObservedRunningTime="2026-10-30 22:05:16.759078862 +0000 UTC m=+1540.709033809" Oct 30 22:05:18.118280 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:05:18.118213 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:05:18.924469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:18.924427 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:18.924683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:18.924501 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:20.950460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:20.950425 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:20.950982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:20.950492 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:22.801688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.801649 5036 generic.go:356] "Generic (PLEG): container finished" podID="66837898-383c-4ac7-93bb-29b5de1a190b" containerID="0216a64334c9859397c9b98badef74818f1d074822cd12d113fb582cde86f208" exitCode=0 Oct 30 22:05:22.802186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.801768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerDied","Data":"0216a64334c9859397c9b98badef74818f1d074822cd12d113fb582cde86f208"} Oct 30 22:05:22.802290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.802187 5036 scope.go:122] "RemoveContainer" containerID="0216a64334c9859397c9b98badef74818f1d074822cd12d113fb582cde86f208" Oct 30 22:05:22.804165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804133 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.804297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804217 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.804297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804283 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.804467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804444 5036 generic.go:356] "Generic (PLEG): container finished" podID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerID="57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e" exitCode=0 Oct 30 22:05:22.804548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804523 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerDied","Data":"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e"} Oct 30 22:05:22.804906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.804891 5036 scope.go:122] "RemoveContainer" containerID="57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e" Oct 30 22:05:22.807008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.806952 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.807104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.807034 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.807104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:22.807086 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:23.892911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.892880 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerStarted","Data":"33435b0726978b771285876da812adc3323c198c02cdf42a20ace5adc2fe53fa"} Oct 30 22:05:23.898301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.898237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerStarted","Data":"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c"} Oct 30 22:05:23.904635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.903471 5036 generic.go:356] "Generic (PLEG): container finished" podID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerID="b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b" exitCode=0 Oct 30 22:05:23.904635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.903653 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerDied","Data":"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b"} Oct 30 22:05:23.904635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.904073 5036 scope.go:122] "RemoveContainer" containerID="b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b" Oct 30 22:05:23.906464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.906185 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:23.906464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.906259 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:23.906464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:23.906315 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:24.457322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.452023 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:24.457322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.452100 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:24.585696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.585648 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:24.602460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.602409 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:24.608751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.608723 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:05:24.609000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:24.608986 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:05:25.396964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:25.392587 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:25.396964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:25.392652 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:25.586198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:25.586167 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:25.586419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:25.586215 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:25.661839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:25.661463 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:05:26.209757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.209724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerStarted","Data":"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801"} Oct 30 22:05:26.390330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.390298 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.390508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.390364 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:26.406895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.406870 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:05:26.407862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.407837 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.407973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.407890 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:26.596494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.596393 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.597246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:26.596802 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:27.401295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.397374 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.401295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.397436 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:27.412350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.410415 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.412350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.410491 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:27.622137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.622050 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.622402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.622372 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:27.710588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.710553 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:05:27.716290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.715449 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:05:27.843314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:27.821907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:28.308333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.306352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:05:28.429288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.409061 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.429913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.429874 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:28.452289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.417766 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.457039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.456496 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:28.639356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.625438 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.668416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.668367 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:28.731499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.731467 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:05:28.733229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.733204 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:05:28.834141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.832400 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:05:28.857384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.857341 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:28.900311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.858730 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:05:28.902019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:28.901962 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:29.257851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.257754 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:05:29.318294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.259336 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:29.384448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.384396 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:29.392295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.388760 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:05:29.476336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.397856 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:29.478950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.476961 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.495964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.479147 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:29.496226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.478573 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.496430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.496392 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:29.590747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.590660 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.591007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.590979 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 22:05:29.618291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.602886 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:05:29.775288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:29.758816 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 30 22:05:30.059974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.059927 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:30.179256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.179203 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:30.308534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.308481 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:05:30.394885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.392160 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 22:05:30.394885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.392215 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 22:05:30.409438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.409403 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:30.414290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.410343 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:30.748795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.748699 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:05:30.753288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.750965 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:05:30.768385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:30.768157 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:05:31.395325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:31.395296 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:05:31.395854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:31.395826 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:05:31.396768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:31.396750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:05:31.407336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:31.407312 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:31.407446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:31.407371 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:32.407583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.407550 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:32.408013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.407601 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:32.567053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.566706 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:05:32.574422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.574395 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.578109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.578081 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:05:32.583846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.583822 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-18ba2f\"/\"openshift-service-ca.crt\"" Oct 30 22:05:32.584210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.584144 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-18ba2f\"/\"kube-root-ca.crt\"" Oct 30 22:05:32.584210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.584203 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-18ba2f\"/\"auth-test-model-18ba2f-kserve-self-signed-certs\"" Oct 30 22:05:32.584597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.584531 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-18ba2f\"/\"default-dockercfg-th7dr\"" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774344 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4brc\" (UniqueName: \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774500 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774603 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.776828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.774848 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.881828 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4brc\" (UniqueName: \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882331 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.882870 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.885301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.883131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.895289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.889807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.992681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.989891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:32.998891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:32.998811 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4brc\" (UniqueName: \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:33.012299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:33.011787 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\") pod \"auth-test-model-18ba2f-kserve-6475786665-gd4jv\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:33.202288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:33.193577 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:33.419323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:33.411087 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.419323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:33.411258 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 22:05:33.775970 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:33.775943 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c0b8b0d_ec77_4a40_afca_9a622c6c6b72.slice/crio-fac6ec776c0fe0111004865a102bf05366002f031e15fec7ee0b22d99ed98d19 WatchSource:0}: Error finding container fac6ec776c0fe0111004865a102bf05366002f031e15fec7ee0b22d99ed98d19: Status 404 returned error can't find the container with id fac6ec776c0fe0111004865a102bf05366002f031e15fec7ee0b22d99ed98d19 Oct 30 22:05:33.776518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:33.776495 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:05:34.039954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.039915 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" event={"ID":"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72","Type":"ContainerStarted","Data":"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183"} Oct 30 22:05:34.040166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.039968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" event={"ID":"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72","Type":"ContainerStarted","Data":"fac6ec776c0fe0111004865a102bf05366002f031e15fec7ee0b22d99ed98d19"} Oct 30 22:05:34.062047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.061945 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" podStartSLOduration=2.06193204 podStartE2EDuration="2.06193204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:34.057948969 +0000 UTC m=+1558.007903914" watchObservedRunningTime="2026-10-30 22:05:34.06193204 +0000 UTC m=+1558.011886985" Oct 30 22:05:34.194766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.194740 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:34.521721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.521692 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:05:34.524881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:34.524856 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:05:37.376392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.376361 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88"] Oct 30 22:05:37.381444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.381425 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.384918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.384894 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:37.385424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.385258 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:05:37.385424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.385365 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:37.385424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.385393 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:37.390473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.390450 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88"] Oct 30 22:05:37.499552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.499513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.499731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.499640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbnr\" (UniqueName: \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.499731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.499664 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.600971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.600937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.601173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.601034 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbnr\" (UniqueName: \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.601173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.601054 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.601734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.601709 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.603655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.603635 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.633341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.622369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbnr\" (UniqueName: \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\") pod \"maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:37.701982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:37.701955 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:38.254683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:38.254651 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88"] Oct 30 22:05:39.170390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:39.170349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" event={"ID":"465d8984-99b2-4d60-9cbe-bf95734bf045","Type":"ContainerStarted","Data":"eb5460b26a3e74ecbd4fd2ab9f41d75c779518920b080604581d62fadcfc5c94"} Oct 30 22:05:39.170887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:39.170444 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" event={"ID":"465d8984-99b2-4d60-9cbe-bf95734bf045","Type":"ContainerStarted","Data":"02224390b5993e6dfbf4b263230b8bf2f46a9e8ecb4a7be496c4416a3b8f8ee1"} Oct 30 22:05:39.189735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:39.189638 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" podStartSLOduration=2.189625394 podStartE2EDuration="2.189625394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:39.187336684 +0000 UTC m=+1563.137291629" watchObservedRunningTime="2026-10-30 22:05:39.189625394 +0000 UTC m=+1563.139580340" Oct 30 22:05:40.182209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:40.182174 5036 generic.go:356] "Generic (PLEG): container finished" podID="465d8984-99b2-4d60-9cbe-bf95734bf045" containerID="eb5460b26a3e74ecbd4fd2ab9f41d75c779518920b080604581d62fadcfc5c94" exitCode=0 Oct 30 22:05:40.190206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:40.190177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" event={"ID":"465d8984-99b2-4d60-9cbe-bf95734bf045","Type":"ContainerDied","Data":"eb5460b26a3e74ecbd4fd2ab9f41d75c779518920b080604581d62fadcfc5c94"} Oct 30 22:05:41.582211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.582179 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:41.727109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.712227 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\") pod \"465d8984-99b2-4d60-9cbe-bf95734bf045\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " Oct 30 22:05:41.727775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.727738 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\") pod \"465d8984-99b2-4d60-9cbe-bf95734bf045\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " Oct 30 22:05:41.728084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.728063 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\" (UniqueName: \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\") pod \"465d8984-99b2-4d60-9cbe-bf95734bf045\" (UID: \"465d8984-99b2-4d60-9cbe-bf95734bf045\") " Oct 30 22:05:41.815658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.799471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca" pod "465d8984-99b2-4d60-9cbe-bf95734bf045" (UID: "465d8984-99b2-4d60-9cbe-bf95734bf045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:41.819448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.819408 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca" pod "465d8984-99b2-4d60-9cbe-bf95734bf045" (UID: "465d8984-99b2-4d60-9cbe-bf95734bf045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:41.836233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.836195 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.836233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.836230 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465d8984-99b2-4d60-9cbe-bf95734bf045-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.846333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.846189 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr" pod "465d8984-99b2-4d60-9cbe-bf95734bf045" (UID: "465d8984-99b2-4d60-9cbe-bf95734bf045"). InnerVolumeSpecName "kube-api-access-ngbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:41.946220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:41.939043 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngbnr\" (UniqueName: \"kubernetes.io/projected/465d8984-99b2-4d60-9cbe-bf95734bf045-kube-api-access-ngbnr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:42.203736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:42.203708 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" Oct 30 22:05:42.203883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:42.203746 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88" event={"ID":"465d8984-99b2-4d60-9cbe-bf95734bf045","Type":"ContainerDied","Data":"02224390b5993e6dfbf4b263230b8bf2f46a9e8ecb4a7be496c4416a3b8f8ee1"} Oct 30 22:05:42.203883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:42.203802 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02224390b5993e6dfbf4b263230b8bf2f46a9e8ecb4a7be496c4416a3b8f8ee1" Oct 30 22:05:43.197288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:43.197010 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:43.325295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:43.323953 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:43.360628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:43.360593 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:05:44.289023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:44.288988 5036 scope.go:122] "RemoveContainer" containerID="adb2383ef084693422561370d2c21bc5f891dfa510f33ac8ae3de7701deb6749" Oct 30 22:05:44.302229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:44.302201 5036 scope.go:122] "RemoveContainer" containerID="e024d41f3a536f794380231ef0e73cf4048ebb47507b78597d8d23d606574921" Oct 30 22:05:46.304323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.304210 5036 generic.go:356] "Generic (PLEG): container finished" podID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerID="aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7" exitCode=0 Oct 30 22:05:46.304323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.304292 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerDied","Data":"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7"} Oct 30 22:05:46.304791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.304702 5036 scope.go:122] "RemoveContainer" containerID="aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7" Oct 30 22:05:46.306413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.306387 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:46.306497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.306455 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:46.306497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:46.306491 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:47.346219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:47.346184 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerStarted","Data":"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b"} Oct 30 22:05:47.778014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:47.777982 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:47.779418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:47.779385 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:47.779638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:47.779609 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:48.355831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:05:48.355725 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:05:48.778305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:48.778229 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:48.778520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:48.778337 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:49.777853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:49.777817 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:49.778306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:49.777871 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:50.778086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:50.778043 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:50.778625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:50.778104 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:51.446141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.446101 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p"] Oct 30 22:05:51.447041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.447016 5036 cpu_manager.go:414] "containerMap: removing container" podUID="465d8984-99b2-4d60-9cbe-bf95734bf045" containerName="revoke-keys" Oct 30 22:05:51.447041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.447039 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465d8984-99b2-4d60-9cbe-bf95734bf045" containerName="revoke-keys" Oct 30 22:05:51.447305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.447256 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="465d8984-99b2-4d60-9cbe-bf95734bf045" containerName="revoke-keys" Oct 30 22:05:51.452312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.452291 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.456334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.456306 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p"] Oct 30 22:05:51.456625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.456599 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:51.456839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.456819 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:51.456929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.456897 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:51.456981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.456921 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:05:51.537880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.537843 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.538162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.538140 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bm9\" (UniqueName: \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.538346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.538325 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.639865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.639827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bm9\" (UniqueName: \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.640100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.639930 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.640100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.640085 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.640810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.640785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.643678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.643654 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.653972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.653940 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bm9\" (UniqueName: \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\") pod \"maas-api-revoke-sub-5491fcbe-thc4p\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:51.721465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.721375 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:51.779068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.779035 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 22:05:51.779754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.779718 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 22:05:51.780606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:51.780580 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:52.145993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:52.145962 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p"] Oct 30 22:05:52.146736 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:52.146711 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699a9764_b4df_4d37_90eb_521d7357a53b.slice/crio-c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9 WatchSource:0}: Error finding container c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9: Status 404 returned error can't find the container with id c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9 Oct 30 22:05:52.498232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:52.498140 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" event={"ID":"699a9764-b4df-4d37-90eb-521d7357a53b","Type":"ContainerStarted","Data":"c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9"} Oct 30 22:05:52.783848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:52.783733 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:52.785562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:52.785536 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:05:53.509206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:53.509175 5036 generic.go:356] "Generic (PLEG): container finished" podID="699a9764-b4df-4d37-90eb-521d7357a53b" containerID="55d7a6d951de0aee9790cc42573565e4a3f69d26e049bba70a1bd39162d17cd7" exitCode=0 Oct 30 22:05:53.509538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:53.509304 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" event={"ID":"699a9764-b4df-4d37-90eb-521d7357a53b","Type":"ContainerDied","Data":"55d7a6d951de0aee9790cc42573565e4a3f69d26e049bba70a1bd39162d17cd7"} Oct 30 22:05:54.764763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.764737 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:54.803781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.803699 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\") pod \"699a9764-b4df-4d37-90eb-521d7357a53b\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " Oct 30 22:05:54.803943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.803781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\") pod \"699a9764-b4df-4d37-90eb-521d7357a53b\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " Oct 30 22:05:54.803943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.803890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\" (UniqueName: \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\") pod \"699a9764-b4df-4d37-90eb-521d7357a53b\" (UID: \"699a9764-b4df-4d37-90eb-521d7357a53b\") " Oct 30 22:05:54.804197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.804168 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca" pod "699a9764-b4df-4d37-90eb-521d7357a53b" (UID: "699a9764-b4df-4d37-90eb-521d7357a53b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:54.804817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.804790 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:54.806629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.806595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9" pod "699a9764-b4df-4d37-90eb-521d7357a53b" (UID: "699a9764-b4df-4d37-90eb-521d7357a53b"). InnerVolumeSpecName "kube-api-access-t2bm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:54.812315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.810209 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca" pod "699a9764-b4df-4d37-90eb-521d7357a53b" (UID: "699a9764-b4df-4d37-90eb-521d7357a53b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:54.906134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.906098 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bm9\" (UniqueName: \"kubernetes.io/projected/699a9764-b4df-4d37-90eb-521d7357a53b-kube-api-access-t2bm9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:54.906134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:54.906130 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a9764-b4df-4d37-90eb-521d7357a53b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:55.531851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:55.531817 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" Oct 30 22:05:55.531851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:55.531837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p" event={"ID":"699a9764-b4df-4d37-90eb-521d7357a53b","Type":"ContainerDied","Data":"c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9"} Oct 30 22:05:55.532091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:55.531899 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ff4383b8d657f34ff92f20f75c2edcca8a781d99d4fbce59e58dce3d79fec9" Oct 30 22:05:58.870390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.870351 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:05:58.871738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.871713 5036 cpu_manager.go:414] "containerMap: removing container" podUID="699a9764-b4df-4d37-90eb-521d7357a53b" containerName="revoke-keys" Oct 30 22:05:58.871841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.871742 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699a9764-b4df-4d37-90eb-521d7357a53b" containerName="revoke-keys" Oct 30 22:05:58.872014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.871998 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="699a9764-b4df-4d37-90eb-521d7357a53b" containerName="revoke-keys" Oct 30 22:05:58.876915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.876890 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.881245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.881220 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-38d6e2\"/\"default-dockercfg-jxlb5\"" Oct 30 22:05:58.881783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.881741 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-38d6e2\"/\"auth-test-model-38d6e2-kserve-self-signed-certs\"" Oct 30 22:05:58.883626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.883599 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:05:58.884534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.884512 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-38d6e2\"/\"openshift-service-ca.crt\"" Oct 30 22:05:58.884653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.884639 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-38d6e2\"/\"kube-root-ca.crt\"" Oct 30 22:05:58.998708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.998678 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.998852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.998754 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.998932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.998917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.998986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.998949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.998986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.998971 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:58.999157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:58.999134 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.106738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.106881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.106959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.107145 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.107235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.107318 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.108097 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.108812 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.111703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.109054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.129109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.121117 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.129617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.129583 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.157843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.157805 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\") pod \"auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.195669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.195637 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:05:59.549948 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:05:59.549918 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37dec2d7_fc83_40ab_b66b_94272db2f376.slice/crio-571212b8698bc4f2bf57429fa81854c8970627fc760f9dec1ddfdd76aded52d1 WatchSource:0}: Error finding container 571212b8698bc4f2bf57429fa81854c8970627fc760f9dec1ddfdd76aded52d1: Status 404 returned error can't find the container with id 571212b8698bc4f2bf57429fa81854c8970627fc760f9dec1ddfdd76aded52d1 Oct 30 22:05:59.550805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.550779 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:05:59.604129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:05:59.604102 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" event={"ID":"37dec2d7-fc83-40ab-b66b-94272db2f376","Type":"ContainerStarted","Data":"571212b8698bc4f2bf57429fa81854c8970627fc760f9dec1ddfdd76aded52d1"} Oct 30 22:06:00.619816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:00.619780 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" event={"ID":"37dec2d7-fc83-40ab-b66b-94272db2f376","Type":"ContainerStarted","Data":"b65e5a002e560f554d6ddc2af5cfe3f89c9da9696fbe5e0d238e684e7235323b"} Oct 30 22:06:00.640959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:00.640913 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" podStartSLOduration=2.640901413 podStartE2EDuration="2.640901413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:00.637330568 +0000 UTC m=+1584.587285531" watchObservedRunningTime="2026-10-30 22:06:00.640901413 +0000 UTC m=+1584.590856359" Oct 30 22:06:01.196148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:01.196113 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:06:08.564707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:08.564644 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88"] Oct 30 22:06:08.606339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:08.602702 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5d81a2-kbk88"] Oct 30 22:06:09.196839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:09.196804 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:06:09.209936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:09.209910 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:06:09.881122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:09.881096 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:06:10.192640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:10.192610 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465d8984-99b2-4d60-9cbe-bf95734bf045" path="/var/lib/kubelet/pods/465d8984-99b2-4d60-9cbe-bf95734bf045/volumes" Oct 30 22:06:13.981450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:13.981414 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:13.981876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:13.981499 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:17.972764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.972732 5036 generic.go:356] "Generic (PLEG): container finished" podID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerID="ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7" exitCode=0 Oct 30 22:06:17.973193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.972799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerDied","Data":"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7"} Oct 30 22:06:17.973362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.973342 5036 scope.go:122] "RemoveContainer" containerID="ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7" Oct 30 22:06:17.975156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.975122 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:17.975254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.975216 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:17.975334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:17.975256 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:18.365983 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:06:18.365950 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:06:19.065332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:19.060015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerStarted","Data":"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2"} Oct 30 22:06:19.829291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:19.825858 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:06:19.829291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:19.826200 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:06:19.829291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:19.826258 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:06:20.827293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:20.827044 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:06:20.827293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:20.827102 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:06:21.828889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:21.828852 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:06:21.829515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:21.829483 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:06:22.828345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:22.828308 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:06:22.828678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:22.828646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:06:23.828824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:23.828790 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 22:06:23.829440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:23.829410 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 22:06:24.280665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.280574 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:06:24.290197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.290165 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.295188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.295154 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:06:24.297711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.297655 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c8ecf\"/\"default-dockercfg-zv5c8\"" Oct 30 22:06:24.297834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.297774 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c8ecf\"/\"auth-test-model-9c8ecf-kserve-self-signed-certs\"" Oct 30 22:06:24.297903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.297831 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c8ecf\"/\"openshift-service-ca.crt\"" Oct 30 22:06:24.299168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.297901 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c8ecf\"/\"kube-root-ca.crt\"" Oct 30 22:06:24.379958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.379925 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.380126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.379966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2lc\" (UniqueName: \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.380126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.380074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.380126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.380110 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.380367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.380350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.380413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.380401 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484350 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2lc\" (UniqueName: \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484554 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.485294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.484861 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.503293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.494025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.525506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.525474 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.525955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.525887 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.527979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.527956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.557362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.547669 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.562051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.561966 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2lc\" (UniqueName: \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\") pod \"auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.627849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.627810 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:24.932479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.930913 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:06:24.934348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.934015 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:06:24.936284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:24.936191 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:06:25.044462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.044432 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:06:25.044659 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:06:25.044633 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade4040b_5424_47dc_b285_5f84ae014ef1.slice/crio-e3917022cdce6c6cfdab3257ed1666dd34d53b6db79e99ec70fe97c2199f3e7b WatchSource:0}: Error finding container e3917022cdce6c6cfdab3257ed1666dd34d53b6db79e99ec70fe97c2199f3e7b: Status 404 returned error can't find the container with id e3917022cdce6c6cfdab3257ed1666dd34d53b6db79e99ec70fe97c2199f3e7b Oct 30 22:06:25.100239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.100206 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg"] Oct 30 22:06:25.113094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.112486 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc186aaf-25gkg"] Oct 30 22:06:25.205022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.203982 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" event={"ID":"ade4040b-5424-47dc-b285-5f84ae014ef1","Type":"ContainerStarted","Data":"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57"} Oct 30 22:06:25.205022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.204034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" event={"ID":"ade4040b-5424-47dc-b285-5f84ae014ef1","Type":"ContainerStarted","Data":"e3917022cdce6c6cfdab3257ed1666dd34d53b6db79e99ec70fe97c2199f3e7b"} Oct 30 22:06:25.231869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.231817 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" podStartSLOduration=1.231800541 podStartE2EDuration="1.231800541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.226460013 +0000 UTC m=+1609.176414963" watchObservedRunningTime="2026-10-30 22:06:25.231800541 +0000 UTC m=+1609.181755490" Oct 30 22:06:25.628436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:25.628354 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:26.190856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:26.190823 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ee8749-ccec-421e-a955-871487de1de1" path="/var/lib/kubelet/pods/b0ee8749-ccec-421e-a955-871487de1de1/volumes" Oct 30 22:06:34.628710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:34.628674 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:34.647321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:34.647255 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:35.355477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:35.355448 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:06:44.371683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:44.371650 5036 scope.go:122] "RemoveContainer" containerID="e9882f7bbee9a98c8d9833dcd30ffa41a831d8ac7249a3debc9073f43f4fcdb0" Oct 30 22:06:44.384560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:44.384538 5036 scope.go:122] "RemoveContainer" containerID="6b7bb52ca635f2e0c1f6dff52f70a6036b654bb0ee16c0607416d1f3c988b513" Oct 30 22:06:44.397544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:44.397526 5036 scope.go:122] "RemoveContainer" containerID="86b4b9f9c1aa90a010f4da39393c56823ab074863ea7d5f6b1d78eee72f89c5a" Oct 30 22:06:48.100957 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:06:48.100918 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:06:59.105331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.105296 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:06:59.105913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.105529 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" containerID="cri-o://6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4" gracePeriod=30 Oct 30 22:06:59.638013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.637981 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx"] Oct 30 22:06:59.646414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.646385 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.650311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.650281 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:59.650464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.650439 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:59.657153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.652727 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx"] Oct 30 22:06:59.675565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.675537 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:59.678904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.675938 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:06:59.787652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.787624 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.787846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.787742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.788010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.787994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrtg\" (UniqueName: \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.889714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.889631 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.889895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.889733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.890053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.890032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrtg\" (UniqueName: \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.890514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.890495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.892516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.892491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.914218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.914194 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrtg\" (UniqueName: \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\") pod \"maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:06:59.992948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:06:59.992917 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:07:00.291448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:00.291411 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx"] Oct 30 22:07:00.795759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:00.795707 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" event={"ID":"ccce58fe-5898-42cd-8176-5d9aeef3f017","Type":"ContainerStarted","Data":"ef1c023c2e47dbbdb8a5972e4cef8ebb3d0e3c4bba80cb7511b2627f6b0a5f77"} Oct 30 22:07:00.795972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:00.795773 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" event={"ID":"ccce58fe-5898-42cd-8176-5d9aeef3f017","Type":"ContainerStarted","Data":"337bd46267b36e37ebc8f88331593fbd0b1edb35e48c553678d2ec16dea07d0e"} Oct 30 22:07:00.817285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:00.817215 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" podStartSLOduration=1.817202318 podStartE2EDuration="1.817202318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:00.812257834 +0000 UTC m=+1644.762212780" watchObservedRunningTime="2026-10-30 22:07:00.817202318 +0000 UTC m=+1644.767157264" Oct 30 22:07:01.811745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:01.811706 5036 generic.go:356] "Generic (PLEG): container finished" podID="ccce58fe-5898-42cd-8176-5d9aeef3f017" containerID="ef1c023c2e47dbbdb8a5972e4cef8ebb3d0e3c4bba80cb7511b2627f6b0a5f77" exitCode=0 Oct 30 22:07:01.812145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:01.811768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" event={"ID":"ccce58fe-5898-42cd-8176-5d9aeef3f017","Type":"ContainerDied","Data":"ef1c023c2e47dbbdb8a5972e4cef8ebb3d0e3c4bba80cb7511b2627f6b0a5f77"} Oct 30 22:07:03.032810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.032782 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:07:03.077667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.077579 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\" (UniqueName: \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\") pod \"ccce58fe-5898-42cd-8176-5d9aeef3f017\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " Oct 30 22:07:03.077836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.077804 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\") pod \"ccce58fe-5898-42cd-8176-5d9aeef3f017\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " Oct 30 22:07:03.077920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.077838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\") pod \"ccce58fe-5898-42cd-8176-5d9aeef3f017\" (UID: \"ccce58fe-5898-42cd-8176-5d9aeef3f017\") " Oct 30 22:07:03.078723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.078682 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca" pod "ccce58fe-5898-42cd-8176-5d9aeef3f017" (UID: "ccce58fe-5898-42cd-8176-5d9aeef3f017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:03.081511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.081481 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca" pod "ccce58fe-5898-42cd-8176-5d9aeef3f017" (UID: "ccce58fe-5898-42cd-8176-5d9aeef3f017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:03.082060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.082022 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg" pod "ccce58fe-5898-42cd-8176-5d9aeef3f017" (UID: "ccce58fe-5898-42cd-8176-5d9aeef3f017"). InnerVolumeSpecName "kube-api-access-5zrtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:03.179991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.179956 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:03.179991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.179993 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccce58fe-5898-42cd-8176-5d9aeef3f017-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:03.180205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.180020 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrtg\" (UniqueName: \"kubernetes.io/projected/ccce58fe-5898-42cd-8176-5d9aeef3f017-kube-api-access-5zrtg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:03.846697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.846652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" event={"ID":"ccce58fe-5898-42cd-8176-5d9aeef3f017","Type":"ContainerDied","Data":"337bd46267b36e37ebc8f88331593fbd0b1edb35e48c553678d2ec16dea07d0e"} Oct 30 22:07:03.846919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.846756 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337bd46267b36e37ebc8f88331593fbd0b1edb35e48c553678d2ec16dea07d0e" Oct 30 22:07:03.846919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:03.846846 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx" Oct 30 22:07:04.413984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.413959 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:07:04.504146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504113 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504151 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504176 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504347 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504387 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504528 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504616 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:04.504863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504807 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\") pod \"5cd355ee-d65d-44b6-9577-514a80a87e1d\" (UID: \"5cd355ee-d65d-44b6-9577-514a80a87e1d\") " Oct 30 22:07:04.504984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504922 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:04.504984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.504951 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:04.505178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505143 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:04.505702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505672 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:04.505822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505730 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd355ee-d65d-44b6-9577-514a80a87e1d-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.505822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505752 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.505822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505767 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.505822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.505780 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.506970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.506944 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:04.507877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.507849 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:04.508240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.508220 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:04.508802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.508774 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp" pod "5cd355ee-d65d-44b6-9577-514a80a87e1d" (UID: "5cd355ee-d65d-44b6-9577-514a80a87e1d"). InnerVolumeSpecName "kube-api-access-c7pdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:04.607301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.607244 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.607301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.607299 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.607496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.607317 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.607496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.607333 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pdp\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-kube-api-access-c7pdp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.607496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.607348 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd355ee-d65d-44b6-9577-514a80a87e1d-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.860028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.859948 5036 generic.go:356] "Generic (PLEG): container finished" podID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerID="6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4" exitCode=0 Oct 30 22:07:04.860028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.860004 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" Oct 30 22:07:04.860028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.860015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerDied","Data":"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4"} Oct 30 22:07:04.860254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.860060 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk" event={"ID":"5cd355ee-d65d-44b6-9577-514a80a87e1d","Type":"ContainerDied","Data":"46fbeb60c0a381d6fd1153d157f3e6997a449491f9a238af84bbe165b2c46abc"} Oct 30 22:07:04.860254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.860088 5036 scope.go:122] "RemoveContainer" containerID="6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4" Oct 30 22:07:04.874328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.874309 5036 scope.go:122] "RemoveContainer" containerID="17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202" Oct 30 22:07:04.887555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.887532 5036 scope.go:122] "RemoveContainer" containerID="6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4" Oct 30 22:07:04.887870 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:07:04.887850 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4\": container with ID starting with 6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4 not found: ID does not exist" containerID="6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4" Oct 30 22:07:04.887961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.887878 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4"} err="failed to get container status \"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4\": rpc error: code = NotFound desc = could not find container \"6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4\": container with ID starting with 6bccf2758d81d0c7bd27780ced6d8324bf5f038366bed3354afda92e0297c6c4 not found: ID does not exist" Oct 30 22:07:04.887961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.887895 5036 scope.go:122] "RemoveContainer" containerID="17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202" Oct 30 22:07:04.888176 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:07:04.888154 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202\": container with ID starting with 17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202 not found: ID does not exist" containerID="17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202" Oct 30 22:07:04.888249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.888182 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202"} err="failed to get container status \"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202\": rpc error: code = NotFound desc = could not find container \"17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202\": container with ID starting with 17612d48bd2dd8e00f14f5b9abfb1180515665f926fec819441b43c9114e8202 not found: ID does not exist" Oct 30 22:07:04.889111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.889089 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:07:04.894613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:04.894589 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-tj2hk"] Oct 30 22:07:06.192674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:06.192640 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" path="/var/lib/kubelet/pods/5cd355ee-d65d-44b6-9577-514a80a87e1d/volumes" Oct 30 22:07:09.825097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:09.825060 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:07:14.827679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:14.827634 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" podUID="29559cc5-4272-495d-ac4a-fca1ac5cd192" containerName="payload-pre-processing" containerID="cri-o://0851017d5c5e3337563887c0701b94164468dcb409d340444e3ff814ddc9a263" gracePeriod=25 Oct 30 22:07:14.976309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:14.976255 5036 generic.go:356] "Generic (PLEG): container finished" podID="29559cc5-4272-495d-ac4a-fca1ac5cd192" containerID="0851017d5c5e3337563887c0701b94164468dcb409d340444e3ff814ddc9a263" exitCode=0 Oct 30 22:07:14.976492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:14.976341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" event={"ID":"29559cc5-4272-495d-ac4a-fca1ac5cd192","Type":"ContainerDied","Data":"0851017d5c5e3337563887c0701b94164468dcb409d340444e3ff814ddc9a263"} Oct 30 22:07:15.155666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.155636 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:07:15.482229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.480775 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\") pod \"29559cc5-4272-495d-ac4a-fca1ac5cd192\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " Oct 30 22:07:15.482229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.480862 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\" (UniqueName: \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\") pod \"29559cc5-4272-495d-ac4a-fca1ac5cd192\" (UID: \"29559cc5-4272-495d-ac4a-fca1ac5cd192\") " Oct 30 22:07:15.482947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.482919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume" pod "29559cc5-4272-495d-ac4a-fca1ac5cd192" (UID: "29559cc5-4272-495d-ac4a-fca1ac5cd192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:15.489997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.489950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk" pod "29559cc5-4272-495d-ac4a-fca1ac5cd192" (UID: "29559cc5-4272-495d-ac4a-fca1ac5cd192"). InnerVolumeSpecName "kube-api-access-nb8qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:15.566991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.566945 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx"] Oct 30 22:07:15.572690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.572663 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2cfa71-lc2bx"] Oct 30 22:07:15.583505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.583470 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29559cc5-4272-495d-ac4a-fca1ac5cd192-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:15.583505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.583499 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb8qk\" (UniqueName: \"kubernetes.io/projected/29559cc5-4272-495d-ac4a-fca1ac5cd192-kube-api-access-nb8qk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:15.989603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.989570 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" Oct 30 22:07:15.989603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.989592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t" event={"ID":"29559cc5-4272-495d-ac4a-fca1ac5cd192","Type":"ContainerDied","Data":"b33c5d47b36bd5595f3bf7516f12644a92d601e662092b051a724f964f538324"} Oct 30 22:07:15.990125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:15.989659 5036 scope.go:122] "RemoveContainer" containerID="0851017d5c5e3337563887c0701b94164468dcb409d340444e3ff814ddc9a263" Oct 30 22:07:16.018589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:16.018561 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:07:16.032095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:16.032066 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2cfa71-79854f6879-qd66t"] Oct 30 22:07:16.191570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:16.191538 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29559cc5-4272-495d-ac4a-fca1ac5cd192" path="/var/lib/kubelet/pods/29559cc5-4272-495d-ac4a-fca1ac5cd192/volumes" Oct 30 22:07:16.192083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:16.192065 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccce58fe-5898-42cd-8176-5d9aeef3f017" path="/var/lib/kubelet/pods/ccce58fe-5898-42cd-8176-5d9aeef3f017/volumes" Oct 30 22:07:18.111253 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:07:18.111209 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:07:25.883515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.883457 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6"] Oct 30 22:07:25.884352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884331 5036 cpu_manager.go:414] "containerMap: removing container" podUID="29559cc5-4272-495d-ac4a-fca1ac5cd192" containerName="payload-pre-processing" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884355 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29559cc5-4272-495d-ac4a-fca1ac5cd192" containerName="payload-pre-processing" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884378 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884384 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884394 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ccce58fe-5898-42cd-8176-5d9aeef3f017" containerName="revoke-keys" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884399 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccce58fe-5898-42cd-8176-5d9aeef3f017" containerName="revoke-keys" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884415 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:25.884443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884419 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:25.884658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884559 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:25.884658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884577 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="29559cc5-4272-495d-ac4a-fca1ac5cd192" containerName="payload-pre-processing" Oct 30 22:07:25.884658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.884585 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccce58fe-5898-42cd-8176-5d9aeef3f017" containerName="revoke-keys" Oct 30 22:07:25.888610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.888593 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:25.897895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.897872 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:25.898519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.898499 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:07:25.899085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.899065 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:25.899085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.899071 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:25.905488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.905463 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6"] Oct 30 22:07:25.963771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.963724 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:25.964378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.964343 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:25.964636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:25.964619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgqz\" (UniqueName: \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.066808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.066780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.066995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.066866 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.066995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.066911 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgqz\" (UniqueName: \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.067590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.067566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.069446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.069426 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.079481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.079458 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgqz\" (UniqueName: \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\") pod \"maas-api-revoke-sub-81503875-w4vz6\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.204740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.204670 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:26.505033 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:07:26.505005 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf52da4_2011_4749_8652_a2a49dd10613.slice/crio-410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654 WatchSource:0}: Error finding container 410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654: Status 404 returned error can't find the container with id 410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654 Oct 30 22:07:26.505491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:26.505466 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6"] Oct 30 22:07:27.222154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:27.222108 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" event={"ID":"ccf52da4-2011-4749-8652-a2a49dd10613","Type":"ContainerStarted","Data":"76cf534339da000b1a2e7dd76daba93615dcef3118adc8b0996a41cd9d811551"} Oct 30 22:07:27.222627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:27.222170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" event={"ID":"ccf52da4-2011-4749-8652-a2a49dd10613","Type":"ContainerStarted","Data":"410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654"} Oct 30 22:07:27.248060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:27.248006 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" podStartSLOduration=2.247993333 podStartE2EDuration="2.247993333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:27.244440035 +0000 UTC m=+1671.194394962" watchObservedRunningTime="2026-10-30 22:07:27.247993333 +0000 UTC m=+1671.197948316" Oct 30 22:07:28.237549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:28.237505 5036 generic.go:356] "Generic (PLEG): container finished" podID="ccf52da4-2011-4749-8652-a2a49dd10613" containerID="76cf534339da000b1a2e7dd76daba93615dcef3118adc8b0996a41cd9d811551" exitCode=0 Oct 30 22:07:28.238309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:28.238189 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" event={"ID":"ccf52da4-2011-4749-8652-a2a49dd10613","Type":"ContainerDied","Data":"76cf534339da000b1a2e7dd76daba93615dcef3118adc8b0996a41cd9d811551"} Oct 30 22:07:29.508790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.508719 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:29.644857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.644817 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\") pod \"ccf52da4-2011-4749-8652-a2a49dd10613\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " Oct 30 22:07:29.645052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.644974 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\" (UniqueName: \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\") pod \"ccf52da4-2011-4749-8652-a2a49dd10613\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " Oct 30 22:07:29.645130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.645120 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\") pod \"ccf52da4-2011-4749-8652-a2a49dd10613\" (UID: \"ccf52da4-2011-4749-8652-a2a49dd10613\") " Oct 30 22:07:29.645600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.645566 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca" pod "ccf52da4-2011-4749-8652-a2a49dd10613" (UID: "ccf52da4-2011-4749-8652-a2a49dd10613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:29.645962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.645942 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:29.647381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.647357 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz" pod "ccf52da4-2011-4749-8652-a2a49dd10613" (UID: "ccf52da4-2011-4749-8652-a2a49dd10613"). InnerVolumeSpecName "kube-api-access-tfgqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:29.647479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.647464 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca" pod "ccf52da4-2011-4749-8652-a2a49dd10613" (UID: "ccf52da4-2011-4749-8652-a2a49dd10613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:29.747322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.747284 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfgqz\" (UniqueName: \"kubernetes.io/projected/ccf52da4-2011-4749-8652-a2a49dd10613-kube-api-access-tfgqz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:29.747322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:29.747323 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf52da4-2011-4749-8652-a2a49dd10613-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:30.028888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.028838 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh"] Oct 30 22:07:30.036585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.036551 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd25d820-7dvgh"] Oct 30 22:07:30.192028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.191990 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fe99f9-983f-4adc-877d-e8c1ac60250e" path="/var/lib/kubelet/pods/61fe99f9-983f-4adc-877d-e8c1ac60250e/volumes" Oct 30 22:07:30.259025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.258990 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" Oct 30 22:07:30.259197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.259030 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6" event={"ID":"ccf52da4-2011-4749-8652-a2a49dd10613","Type":"ContainerDied","Data":"410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654"} Oct 30 22:07:30.259197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:30.259086 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410b36bda1e502534a984572bf392eba42a9c1cb139e3284590a02dc2cc73654" Oct 30 22:07:36.198126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.198094 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5"] Oct 30 22:07:36.198887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.198868 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ccf52da4-2011-4749-8652-a2a49dd10613" containerName="revoke-keys" Oct 30 22:07:36.198887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.198884 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf52da4-2011-4749-8652-a2a49dd10613" containerName="revoke-keys" Oct 30 22:07:36.199059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.199003 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd355ee-d65d-44b6-9577-514a80a87e1d" containerName="istio-proxy" Oct 30 22:07:36.199059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.199017 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf52da4-2011-4749-8652-a2a49dd10613" containerName="revoke-keys" Oct 30 22:07:36.205142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.205105 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.210764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.210738 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:36.211082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.211063 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:36.211193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.211072 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:36.211193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.211120 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:07:36.212085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.212066 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5"] Oct 30 22:07:36.294954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.294923 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7sr\" (UniqueName: \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.295215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.295194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.295348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.295312 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.399564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.399523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.399750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.399619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.399812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.399771 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7sr\" (UniqueName: \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.401172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.401140 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.462618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.451821 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.462618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.460162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7sr\" (UniqueName: \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\") pod \"maas-api-revoke-sub-1310d2dc-vbmf5\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.548497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.548461 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:36.853950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:36.853919 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5"] Oct 30 22:07:36.854996 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:07:36.854966 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf164e9d1_b7c1_4cd9_ba08_8357ee967694.slice/crio-70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87 WatchSource:0}: Error finding container 70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87: Status 404 returned error can't find the container with id 70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87 Oct 30 22:07:37.479488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:37.479455 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" event={"ID":"f164e9d1-b7c1-4cd9-ba08-8357ee967694","Type":"ContainerStarted","Data":"bcc82fe8b20c3232deec29da3f98a0ef95da0596dc0951673b35beb8355daa81"} Oct 30 22:07:37.479887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:37.479509 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" event={"ID":"f164e9d1-b7c1-4cd9-ba08-8357ee967694","Type":"ContainerStarted","Data":"70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87"} Oct 30 22:07:37.501652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:37.501593 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" podStartSLOduration=1.501576981 podStartE2EDuration="1.501576981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:37.495437421 +0000 UTC m=+1681.445392380" watchObservedRunningTime="2026-10-30 22:07:37.501576981 +0000 UTC m=+1681.451531927" Oct 30 22:07:38.492472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:38.492429 5036 generic.go:356] "Generic (PLEG): container finished" podID="f164e9d1-b7c1-4cd9-ba08-8357ee967694" containerID="bcc82fe8b20c3232deec29da3f98a0ef95da0596dc0951673b35beb8355daa81" exitCode=0 Oct 30 22:07:38.492897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:38.492515 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" event={"ID":"f164e9d1-b7c1-4cd9-ba08-8357ee967694","Type":"ContainerDied","Data":"bcc82fe8b20c3232deec29da3f98a0ef95da0596dc0951673b35beb8355daa81"} Oct 30 22:07:39.743900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.743869 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:39.862852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.862772 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\") pod \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " Oct 30 22:07:39.863019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.862886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\" (UniqueName: \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\") pod \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " Oct 30 22:07:39.863093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.863064 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\") pod \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\" (UID: \"f164e9d1-b7c1-4cd9-ba08-8357ee967694\") " Oct 30 22:07:39.863188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.863166 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca" pod "f164e9d1-b7c1-4cd9-ba08-8357ee967694" (UID: "f164e9d1-b7c1-4cd9-ba08-8357ee967694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:39.863769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.863725 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:39.865199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.865175 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca" pod "f164e9d1-b7c1-4cd9-ba08-8357ee967694" (UID: "f164e9d1-b7c1-4cd9-ba08-8357ee967694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:39.865323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.865220 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr" pod "f164e9d1-b7c1-4cd9-ba08-8357ee967694" (UID: "f164e9d1-b7c1-4cd9-ba08-8357ee967694"). InnerVolumeSpecName "kube-api-access-jg7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:39.965435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.965404 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7sr\" (UniqueName: \"kubernetes.io/projected/f164e9d1-b7c1-4cd9-ba08-8357ee967694-kube-api-access-jg7sr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:39.965435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:39.965433 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f164e9d1-b7c1-4cd9-ba08-8357ee967694-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:40.556594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:40.556506 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" Oct 30 22:07:40.556594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:40.556464 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5" event={"ID":"f164e9d1-b7c1-4cd9-ba08-8357ee967694","Type":"ContainerDied","Data":"70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87"} Oct 30 22:07:40.556845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:40.556821 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70fc94a399358fec6f5ac0269d175931835a25b3e83e9cfb7e1da1a9f1c9fa87" Oct 30 22:07:43.031095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:43.031057 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8"] Oct 30 22:07:43.039356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:43.039329 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9336f746-d8hr8"] Oct 30 22:07:44.218395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.217596 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e10802-7f82-4a19-992f-9668811d03dd" path="/var/lib/kubelet/pods/b1e10802-7f82-4a19-992f-9668811d03dd/volumes" Oct 30 22:07:44.474909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.474818 5036 scope.go:122] "RemoveContainer" containerID="ef99dea0786344599ed578f826f87af9e7efd3b1db481bb7ff3cde03211add92" Oct 30 22:07:44.520372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.520329 5036 scope.go:122] "RemoveContainer" containerID="9295ca65dc686fb4c06c41c318948c98c4809d2565f33bad090572ff6a67ced8" Oct 30 22:07:44.550756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.550568 5036 scope.go:122] "RemoveContainer" containerID="63882080d85ca2256b9f170680d57ba945a4a8dc9cf8a7ca8dc99644ccdd36a3" Oct 30 22:07:44.570766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.570739 5036 scope.go:122] "RemoveContainer" containerID="7dc1741b858bf37200ce4e2948a7bacb62e14f487b8d2edd3d7ae8d36749badf" Oct 30 22:07:44.585774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.585747 5036 scope.go:122] "RemoveContainer" containerID="8bde832068a1bbfe21c10f39f71ba804b945033014cdf036b182905e0f6eb12e" Oct 30 22:07:44.600197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:44.600176 5036 scope.go:122] "RemoveContainer" containerID="546f18ce657a6f1d90af02f496b40f5a8c7f0729231ca2a24925ce3b9bcbeed9" Oct 30 22:07:48.116071 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:07:48.116032 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:07:53.763000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.762965 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:07:53.763942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.763923 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f164e9d1-b7c1-4cd9-ba08-8357ee967694" containerName="revoke-keys" Oct 30 22:07:53.763942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.763944 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f164e9d1-b7c1-4cd9-ba08-8357ee967694" containerName="revoke-keys" Oct 30 22:07:53.764095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.764086 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f164e9d1-b7c1-4cd9-ba08-8357ee967694" containerName="revoke-keys" Oct 30 22:07:53.769966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.769947 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:53.775829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.775805 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0f1f9c-dockercfg-kd8zs\"" Oct 30 22:07:53.775987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.775806 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0f1f9c\"" Oct 30 22:07:53.779318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.779295 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:07:53.888942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.885118 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xhv\" (UniqueName: \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:53.888942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.885195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:53.995290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.994507 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xhv\" (UniqueName: \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:53.995290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.994601 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:53.999287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:53.995714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:54.093335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:54.092400 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xhv\" (UniqueName: \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\") pod \"payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:54.388636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:54.388610 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:54.697822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:54.697794 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:07:54.698065 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:07:54.698046 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0200b64b_c9ee_4e6e_847d_5e05f075f18f.slice/crio-739967ffd25f3190b1caae6b264310ffadbc9fe476e770f587149fd6a88b8915 WatchSource:0}: Error finding container 739967ffd25f3190b1caae6b264310ffadbc9fe476e770f587149fd6a88b8915: Status 404 returned error can't find the container with id 739967ffd25f3190b1caae6b264310ffadbc9fe476e770f587149fd6a88b8915 Oct 30 22:07:54.777487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:54.777452 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" event={"ID":"0200b64b-c9ee-4e6e-847d-5e05f075f18f","Type":"ContainerStarted","Data":"739967ffd25f3190b1caae6b264310ffadbc9fe476e770f587149fd6a88b8915"} Oct 30 22:07:55.791590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:55.791551 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" event={"ID":"0200b64b-c9ee-4e6e-847d-5e05f075f18f","Type":"ContainerStarted","Data":"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e"} Oct 30 22:07:55.792128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:55.791854 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:07:55.813095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:55.813048 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" podStartSLOduration=2.813035234 podStartE2EDuration="2.813035234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:55.808755349 +0000 UTC m=+1699.758710295" watchObservedRunningTime="2026-10-30 22:07:55.813035234 +0000 UTC m=+1699.762990524" Oct 30 22:07:57.035664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.035626 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf"] Oct 30 22:07:57.044552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.044521 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d6381-nqbmf"] Oct 30 22:07:57.880322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.880238 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:07:57.888974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.888943 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.904691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.900847 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:07:57.906131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.906106 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:57.911939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.911913 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:07:57.934599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.934575 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:07:57.981842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.981804 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.981988 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982168 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982246 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982403 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt488\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982490 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:57.982696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:57.982597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.084431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.084431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084454 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084484 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084786 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084811 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcbw\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.084990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085192 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085202 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085256 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085290 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085403 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085438 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.085908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085509 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt488\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085636 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.085908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.085713 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.087980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.087957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.088299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.088259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.097596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.097572 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.097732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.097715 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt488\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.187037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187042 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187089 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187158 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187209 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187380 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187428 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187446 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcbw\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187490 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187586 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.187673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187625 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.188028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.187995 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.188167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.188037 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.190116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.190096 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.190407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.190388 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.191864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.191841 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9f0f1e-2d1c-4b18-ad84-606071208d9a" path="/var/lib/kubelet/pods/ac9f0f1e-2d1c-4b18-ad84-606071208d9a/volumes" Oct 30 22:07:58.199915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.199892 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.200017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.199965 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcbw\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.215168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.215139 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:58.228796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.228719 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:58.530109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.530061 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:07:58.533393 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:07:58.533359 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4824aafc_23a1_4d71_9318_ba472a4f044e.slice/crio-ab9be92e1dce9ff99df96cfcf0e1391f633b20f506709c92601c71129eae7af2 WatchSource:0}: Error finding container ab9be92e1dce9ff99df96cfcf0e1391f633b20f506709c92601c71129eae7af2: Status 404 returned error can't find the container with id ab9be92e1dce9ff99df96cfcf0e1391f633b20f506709c92601c71129eae7af2 Oct 30 22:07:58.536577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.536538 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.536690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.536631 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.536690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.536676 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.551926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.551899 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:07:58.552845 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:07:58.552819 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b70e666_6ed9_4677_8a0b_6598bbd4c28c.slice/crio-9c0b988cee0374da5fde4c1aa8fe25300d3e5052b1b5f2aa81cd0204ddcd0d56 WatchSource:0}: Error finding container 9c0b988cee0374da5fde4c1aa8fe25300d3e5052b1b5f2aa81cd0204ddcd0d56: Status 404 returned error can't find the container with id 9c0b988cee0374da5fde4c1aa8fe25300d3e5052b1b5f2aa81cd0204ddcd0d56 Oct 30 22:07:58.556609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.556560 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.556716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.556648 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.556716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.556693 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:58.831389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.828482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" event={"ID":"2b70e666-6ed9-4677-8a0b-6598bbd4c28c","Type":"ContainerStarted","Data":"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9"} Oct 30 22:07:58.831389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.828549 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" event={"ID":"2b70e666-6ed9-4677-8a0b-6598bbd4c28c","Type":"ContainerStarted","Data":"9c0b988cee0374da5fde4c1aa8fe25300d3e5052b1b5f2aa81cd0204ddcd0d56"} Oct 30 22:07:58.831657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.831417 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" event={"ID":"4824aafc-23a1-4d71-9318-ba472a4f044e","Type":"ContainerStarted","Data":"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc"} Oct 30 22:07:58.831657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.831469 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" event={"ID":"4824aafc-23a1-4d71-9318-ba472a4f044e","Type":"ContainerStarted","Data":"ab9be92e1dce9ff99df96cfcf0e1391f633b20f506709c92601c71129eae7af2"} Oct 30 22:07:58.857759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.857713 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podStartSLOduration=1.8577001 podStartE2EDuration="1.8577001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:58.852091779 +0000 UTC m=+1702.802046730" watchObservedRunningTime="2026-10-30 22:07:58.8577001 +0000 UTC m=+1702.807655052" Oct 30 22:07:58.880309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:58.880200 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podStartSLOduration=1.880183234 podStartE2EDuration="1.880183234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:58.875181486 +0000 UTC m=+1702.825136457" watchObservedRunningTime="2026-10-30 22:07:58.880183234 +0000 UTC m=+1702.830138182" Oct 30 22:07:59.216089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.216058 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:07:59.217386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.217355 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:07:59.217520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.217422 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:07:59.229015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.228991 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:07:59.230742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.230715 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:07:59.230856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:07:59.230767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:00.225289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:00.218667 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.225289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:00.218725 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:00.251292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:00.238028 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.251292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:00.238091 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:01.229321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:01.219779 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.229321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:01.219842 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:01.233289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:01.233187 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.233289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:01.233241 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:02.215674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:02.215638 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:02.216083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:02.216052 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:02.230661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:02.230629 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:02.231224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:02.231195 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:03.231291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:03.224346 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:03.231291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:03.224404 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:03.252299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:03.251688 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:03.252299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:03.251741 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:04.227290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:04.224760 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:04.227290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:04.224820 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:04.242294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:04.235553 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:04.242294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:04.235604 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:05.129293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.128989 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw"] Oct 30 22:08:05.209332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.209173 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a077c-vn5vw"] Oct 30 22:08:05.222010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.221568 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:05.222010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.221629 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:05.232235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.232131 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:05.232235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:05.232183 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:06.197182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.197139 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c2b547-cf90-4e2f-9e6c-ad90ed993d8b" path="/var/lib/kubelet/pods/72c2b547-cf90-4e2f-9e6c-ad90ed993d8b/volumes" Oct 30 22:08:06.215714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.215677 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 22:08:06.215992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.215962 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 22:08:06.229115 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.229083 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 22:08:06.229398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.229370 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 22:08:06.805492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:06.805459 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:08:07.734936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.734891 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:08:07.736589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.736525 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:08:07.738526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.738421 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:08:07.747181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.747008 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:08:07.747738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.747722 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:08:07.748990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:07.748972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:08:09.078795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:09.078758 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w"] Oct 30 22:08:09.088153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:09.088123 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773950ef-8mp9w"] Oct 30 22:08:10.201874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:10.201839 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f65470-ee9b-42c5-a746-53c08ba3aed1" path="/var/lib/kubelet/pods/32f65470-ee9b-42c5-a746-53c08ba3aed1/volumes" Oct 30 22:08:15.084938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:15.084893 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt"] Oct 30 22:08:15.093405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:15.093375 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62af3079-mndnt"] Oct 30 22:08:16.191931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:16.191900 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5617cee-15cb-4ca7-8c8b-e794bc72f5d7" path="/var/lib/kubelet/pods/f5617cee-15cb-4ca7-8c8b-e794bc72f5d7/volumes" Oct 30 22:08:18.102690 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:08:18.102651 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:08:22.042134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:22.042099 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9"] Oct 30 22:08:22.049554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:22.049525 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16dde3ef-8j6g9"] Oct 30 22:08:22.191104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:22.191054 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f87aff-9158-4c40-80d4-fe180ea44e6a" path="/var/lib/kubelet/pods/53f87aff-9158-4c40-80d4-fe180ea44e6a/volumes" Oct 30 22:08:28.029314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:28.029247 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4"] Oct 30 22:08:28.037806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:28.037773 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86706870-68xc4"] Oct 30 22:08:28.190641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:28.190614 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e275ca73-2729-4189-9572-cfdd6dc565a7" path="/var/lib/kubelet/pods/e275ca73-2729-4189-9572-cfdd6dc565a7/volumes" Oct 30 22:08:31.027418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.027374 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp"] Oct 30 22:08:31.033048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.033019 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fc26448-qrqbp"] Oct 30 22:08:31.673370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.673331 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h"] Oct 30 22:08:31.701794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.701763 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h"] Oct 30 22:08:31.701994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.701882 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.708616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.708591 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:31.708933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.708913 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:08:31.709690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.709669 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:31.709885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.709871 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:31.729809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.729590 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.730755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.730710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.731084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.731063 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqk8\" (UniqueName: \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.833525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.833485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.833727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.833635 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.833806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.833742 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqk8\" (UniqueName: \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.834605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.834574 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.836346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.836315 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:31.845720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:31.845695 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqk8\" (UniqueName: \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\") pod \"maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:32.050469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:32.050381 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:32.194069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:32.194032 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921f2b44-6917-481c-8a63-6d494c880b8b" path="/var/lib/kubelet/pods/921f2b44-6917-481c-8a63-6d494c880b8b/volumes" Oct 30 22:08:32.347151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:32.347119 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h"] Oct 30 22:08:32.348115 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:08:32.348087 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5526175_f9a7_488d_b50b_5b4c777d1f5d.slice/crio-c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2 WatchSource:0}: Error finding container c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2: Status 404 returned error can't find the container with id c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2 Oct 30 22:08:32.652548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:32.652506 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" event={"ID":"b5526175-f9a7-488d-b50b-5b4c777d1f5d","Type":"ContainerStarted","Data":"c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2"} Oct 30 22:08:33.680072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:33.679982 5036 generic.go:356] "Generic (PLEG): container finished" podID="b5526175-f9a7-488d-b50b-5b4c777d1f5d" containerID="df03d8ab0d165277f94ec9b903450776bad07525e618d0edd278587122787425" exitCode=0 Oct 30 22:08:33.680629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:33.680324 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" event={"ID":"b5526175-f9a7-488d-b50b-5b4c777d1f5d","Type":"ContainerDied","Data":"df03d8ab0d165277f94ec9b903450776bad07525e618d0edd278587122787425"} Oct 30 22:08:34.890444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:34.890420 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:35.002832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.002731 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\") pod \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " Oct 30 22:08:35.003013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.002874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\") pod \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " Oct 30 22:08:35.003013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.002934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\" (UniqueName: \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\") pod \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\" (UID: \"b5526175-f9a7-488d-b50b-5b4c777d1f5d\") " Oct 30 22:08:35.003383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.003349 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca" pod "b5526175-f9a7-488d-b50b-5b4c777d1f5d" (UID: "b5526175-f9a7-488d-b50b-5b4c777d1f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:35.003962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.003938 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.005400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.005375 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca" pod "b5526175-f9a7-488d-b50b-5b4c777d1f5d" (UID: "b5526175-f9a7-488d-b50b-5b4c777d1f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:35.005825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.005803 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8" pod "b5526175-f9a7-488d-b50b-5b4c777d1f5d" (UID: "b5526175-f9a7-488d-b50b-5b4c777d1f5d"). InnerVolumeSpecName "kube-api-access-cbqk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:35.107887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.107838 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqk8\" (UniqueName: \"kubernetes.io/projected/b5526175-f9a7-488d-b50b-5b4c777d1f5d-kube-api-access-cbqk8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.107887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.107879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5526175-f9a7-488d-b50b-5b4c777d1f5d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.700414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.700382 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" Oct 30 22:08:35.700414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.700402 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h" event={"ID":"b5526175-f9a7-488d-b50b-5b4c777d1f5d","Type":"ContainerDied","Data":"c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2"} Oct 30 22:08:35.700657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:35.700470 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65c696488c1d37dc6e57230757e77e5b378274b737aa89a3048d533f6d53ff2" Oct 30 22:08:40.044785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:40.044736 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv"] Oct 30 22:08:40.050797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:40.050766 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6e1577-gfksv"] Oct 30 22:08:40.190810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:40.190775 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c972105-5945-44d6-81d4-abe096bbbd16" path="/var/lib/kubelet/pods/4c972105-5945-44d6-81d4-abe096bbbd16/volumes" Oct 30 22:08:41.792859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:41.792827 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:08:44.756316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.746953 5036 scope.go:122] "RemoveContainer" containerID="51f1fd278b3876a4f23890c00498cf1ac78720030631d08803bd732832b8778d" Oct 30 22:08:44.808060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.808029 5036 scope.go:122] "RemoveContainer" containerID="0b529d394744529aa69fc45670b6e89bf74b92e0570318bffe483e21bd405da6" Oct 30 22:08:44.821872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.821841 5036 scope.go:122] "RemoveContainer" containerID="12ab4d3e6f5702ed3253f6c6c2000de795b603ac784592ec8ed48412e3222c06" Oct 30 22:08:44.840061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.840037 5036 scope.go:122] "RemoveContainer" containerID="69756596d30707c7b14b51d1efe64e6b7264a1385a30849657032dfb31cfbdf6" Oct 30 22:08:44.855743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.855719 5036 scope.go:122] "RemoveContainer" containerID="00e0fe4cc506a744e3fde98ca6148bbca91ef624f5d6fb0cd6d962088a558645" Oct 30 22:08:44.866704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.866686 5036 scope.go:122] "RemoveContainer" containerID="6889a0e914efd708c11f939ba191009223ffb9018982781c0201b0d8c72b05e4" Oct 30 22:08:44.880313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:44.880290 5036 scope.go:122] "RemoveContainer" containerID="b4e97a2ef2e5bf5179ee1ea8ddf71b4dba66ff274a73c3982b4c989545b44a04" Oct 30 22:08:46.030922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.030823 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl"] Oct 30 22:08:46.038433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.038406 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc"] Oct 30 22:08:46.045801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.045770 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e322c572-f6ntl"] Oct 30 22:08:46.051368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.051344 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e2146-h2rnc"] Oct 30 22:08:46.194366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.194337 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bc0838-3a30-4647-8da6-1fa7b2199f14" path="/var/lib/kubelet/pods/29bc0838-3a30-4647-8da6-1fa7b2199f14/volumes" Oct 30 22:08:46.194832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.194818 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530b98ec-ba52-4908-bfac-8016bb83cf5f" path="/var/lib/kubelet/pods/530b98ec-ba52-4908-bfac-8016bb83cf5f/volumes" Oct 30 22:08:46.802290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:46.801351 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerName="payload-pre-processing" containerID="cri-o://d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e" gracePeriod=25 Oct 30 22:08:46.815353 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:08:46.809224 5036 logging.go:55] [core] [Channel #5279 SubChannel #5280]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.10:50052", ServerName: "10.132.1.10:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.10:50052: connect: connection refused" Oct 30 22:08:47.118400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.118355 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h"] Oct 30 22:08:47.127063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.127029 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f1f9c-zsq9h"] Oct 30 22:08:47.319763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.319733 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:08:47.435300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.432183 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\" (UniqueName: \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\") pod \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " Oct 30 22:08:47.435300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.432252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\") pod \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\" (UID: \"0200b64b-c9ee-4e6e-847d-5e05f075f18f\") " Oct 30 22:08:47.453349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.438624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume" pod "0200b64b-c9ee-4e6e-847d-5e05f075f18f" (UID: "0200b64b-c9ee-4e6e-847d-5e05f075f18f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:47.494289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.492491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv" pod "0200b64b-c9ee-4e6e-847d-5e05f075f18f" (UID: "0200b64b-c9ee-4e6e-847d-5e05f075f18f"). InnerVolumeSpecName "kube-api-access-69xhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:47.534582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.534549 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xhv\" (UniqueName: \"kubernetes.io/projected/0200b64b-c9ee-4e6e-847d-5e05f075f18f-kube-api-access-69xhv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.534582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.534575 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0200b64b-c9ee-4e6e-847d-5e05f075f18f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.809557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:47.809474 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.10:50052\" within 1s: context deadline exceeded" Oct 30 22:08:47.809714 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:08:47.809577 5036 logging.go:55] [core] [Channel #5279 SubChannel #5280]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.10:50052", ServerName: "10.132.1.10:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.10:50052: operation was canceled" Oct 30 22:08:48.095032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.094139 5036 generic.go:356] "Generic (PLEG): container finished" podID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerID="d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e" exitCode=0 Oct 30 22:08:48.095032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.094188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" event={"ID":"0200b64b-c9ee-4e6e-847d-5e05f075f18f","Type":"ContainerDied","Data":"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e"} Oct 30 22:08:48.095032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.094284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" event={"ID":"0200b64b-c9ee-4e6e-847d-5e05f075f18f","Type":"ContainerDied","Data":"739967ffd25f3190b1caae6b264310ffadbc9fe476e770f587149fd6a88b8915"} Oct 30 22:08:48.095032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.094317 5036 scope.go:122] "RemoveContainer" containerID="d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e" Oct 30 22:08:48.095032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.094494 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd" Oct 30 22:08:48.112856 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:08:48.112816 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:08:48.113822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.113795 5036 scope.go:122] "RemoveContainer" containerID="d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e" Oct 30 22:08:48.114153 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:08:48.114127 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e\": container with ID starting with d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e not found: ID does not exist" containerID="d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e" Oct 30 22:08:48.114255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.114164 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e"} err="failed to get container status \"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e\": rpc error: code = NotFound desc = could not find container \"d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e\": container with ID starting with d4f745585bf55cf61fecd2d68ac56d4a9c2d633acab452d5122a7c853e745c1e not found: ID does not exist" Oct 30 22:08:48.133220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.133191 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:08:48.139787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.139766 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f1f9c-5c87fbbf7f-nldkd"] Oct 30 22:08:48.194233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.194197 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" path="/var/lib/kubelet/pods/0200b64b-c9ee-4e6e-847d-5e05f075f18f/volumes" Oct 30 22:08:48.194826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:48.194800 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5526175-f9a7-488d-b50b-5b4c777d1f5d" path="/var/lib/kubelet/pods/b5526175-f9a7-488d-b50b-5b4c777d1f5d/volumes" Oct 30 22:08:49.030069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:49.030035 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm"] Oct 30 22:08:49.037533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:49.037507 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d394678-bfnnm"] Oct 30 22:08:50.191360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:08:50.191329 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717fd2a1-8a3c-48d2-ba92-8dd42fd295bb" path="/var/lib/kubelet/pods/717fd2a1-8a3c-48d2-ba92-8dd42fd295bb/volumes" Oct 30 22:09:02.031466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:02.031426 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42"] Oct 30 22:09:02.036951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:02.036923 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd06d2e-46r42"] Oct 30 22:09:02.190667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:02.190638 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a00178-7496-47fb-96a0-1bfcef277fa3" path="/var/lib/kubelet/pods/01a00178-7496-47fb-96a0-1bfcef277fa3/volumes" Oct 30 22:09:03.130520 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:09:03.130490 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Oct 30 22:09:07.067297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:07.065475 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph"] Oct 30 22:09:07.096290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:07.090756 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b563f2a-2vdph"] Oct 30 22:09:08.032030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:08.031997 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp"] Oct 30 22:09:08.037280 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:08.037235 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeb3c08-k9mjp"] Oct 30 22:09:08.190592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:08.190561 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ea7c9f-b080-400d-82d5-c7b52ed302e6" path="/var/lib/kubelet/pods/85ea7c9f-b080-400d-82d5-c7b52ed302e6/volumes" Oct 30 22:09:08.191104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:08.191087 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fdf083-ed02-4a9c-9588-e33cb2e134b9" path="/var/lib/kubelet/pods/e4fdf083-ed02-4a9c-9588-e33cb2e134b9/volumes" Oct 30 22:09:10.079649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:10.079619 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8"] Oct 30 22:09:10.557293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:10.554034 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2040ee18-48rd8"] Oct 30 22:09:12.192621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:12.192581 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6af612b-f7ce-427a-b4a8-4d550a961a85" path="/var/lib/kubelet/pods/e6af612b-f7ce-427a-b4a8-4d550a961a85/volumes" Oct 30 22:09:13.238392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.238358 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8"] Oct 30 22:09:13.240236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240204 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerName="payload-pre-processing" Oct 30 22:09:13.240236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240236 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerName="payload-pre-processing" Oct 30 22:09:13.240599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240506 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b5526175-f9a7-488d-b50b-5b4c777d1f5d" containerName="revoke-keys" Oct 30 22:09:13.240599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240520 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5526175-f9a7-488d-b50b-5b4c777d1f5d" containerName="revoke-keys" Oct 30 22:09:13.240731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240712 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0200b64b-c9ee-4e6e-847d-5e05f075f18f" containerName="payload-pre-processing" Oct 30 22:09:13.240790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.240748 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5526175-f9a7-488d-b50b-5b4c777d1f5d" containerName="revoke-keys" Oct 30 22:09:13.253888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.253542 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.259170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.259136 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:13.259369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.259305 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:09:13.259369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.259315 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8"] Oct 30 22:09:13.259369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.259349 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:13.259507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.259420 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:13.382568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.382525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.382774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.382665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.382841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.382783 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227ph\" (UniqueName: \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.484173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.484130 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227ph\" (UniqueName: \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.484452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.484415 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.484569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.484554 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.485210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.485183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.487156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.487125 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.496490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.496436 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227ph\" (UniqueName: \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\") pod \"maas-api-revoke-sub-5926cbd4-nrxq8\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.574625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.574581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:13.880641 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:09:13.880607 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824fcfe5_9e3b_4fbf_80ba_5db3f5c502ed.slice/crio-5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c WatchSource:0}: Error finding container 5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c: Status 404 returned error can't find the container with id 5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c Oct 30 22:09:13.881487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:13.881462 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8"] Oct 30 22:09:14.547564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:14.547526 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" event={"ID":"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed","Type":"ContainerStarted","Data":"432c0df8b51555ce467200a316fc48a6907316481e07c0f302e7b5a09c7d4006"} Oct 30 22:09:14.547933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:14.547593 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" event={"ID":"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed","Type":"ContainerStarted","Data":"5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c"} Oct 30 22:09:14.568185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:14.568138 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" podStartSLOduration=1.568125233 podStartE2EDuration="1.568125233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:14.564708397 +0000 UTC m=+1778.514663341" watchObservedRunningTime="2026-10-30 22:09:14.568125233 +0000 UTC m=+1778.518080212" Oct 30 22:09:15.039502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:15.039474 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp"] Oct 30 22:09:15.045734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:15.045708 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-234d5da1-qfbbp"] Oct 30 22:09:15.592108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:15.591474 5036 generic.go:356] "Generic (PLEG): container finished" podID="824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" containerID="432c0df8b51555ce467200a316fc48a6907316481e07c0f302e7b5a09c7d4006" exitCode=0 Oct 30 22:09:15.592108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:15.591738 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" event={"ID":"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed","Type":"ContainerDied","Data":"432c0df8b51555ce467200a316fc48a6907316481e07c0f302e7b5a09c7d4006"} Oct 30 22:09:16.193636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.193602 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5" path="/var/lib/kubelet/pods/ab4dbf5a-3c29-40cf-93cf-a3e5baeb36b5/volumes" Oct 30 22:09:16.827802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.827771 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:16.965926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.965885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\" (UniqueName: \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\") pod \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " Oct 30 22:09:16.966126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.965970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\") pod \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " Oct 30 22:09:16.966126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.966055 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\") pod \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\" (UID: \"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed\") " Oct 30 22:09:16.966464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.966439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca" pod "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" (UID: "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:16.967564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.967534 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.968525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.968498 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph" pod "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" (UID: "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed"). InnerVolumeSpecName "kube-api-access-227ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.968622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:16.968528 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca" pod "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" (UID: "824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:17.087304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:17.086497 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227ph\" (UniqueName: \"kubernetes.io/projected/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-kube-api-access-227ph\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:17.087304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:17.086543 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:17.615376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:17.615337 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" Oct 30 22:09:17.615376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:17.615363 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8" event={"ID":"824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed","Type":"ContainerDied","Data":"5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c"} Oct 30 22:09:17.615653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:17.615411 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d720f67c3fd14f1cf7240172942ec05597b6b168eb1d1754e054d655756f37c" Oct 30 22:09:18.101717 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:09:18.101680 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:09:21.057289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:21.056321 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm"] Oct 30 22:09:21.070291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:21.065470 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4c4750b-h46nm"] Oct 30 22:09:22.072114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:22.072081 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh"] Oct 30 22:09:22.079957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:22.079932 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-351f7b5b-xtprh"] Oct 30 22:09:22.193056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:22.193025 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530a7381-cfac-4a3b-8047-ae21e777a5f2" path="/var/lib/kubelet/pods/530a7381-cfac-4a3b-8047-ae21e777a5f2/volumes" Oct 30 22:09:22.193554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:22.193535 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8609cb16-f71b-4202-9c52-7165cc1bc49c" path="/var/lib/kubelet/pods/8609cb16-f71b-4202-9c52-7165cc1bc49c/volumes" Oct 30 22:09:23.426517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.426484 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk"] Oct 30 22:09:23.427321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.427277 5036 cpu_manager.go:414] "containerMap: removing container" podUID="824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" containerName="revoke-keys" Oct 30 22:09:23.427321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.427297 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" containerName="revoke-keys" Oct 30 22:09:23.427511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.427463 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" containerName="revoke-keys" Oct 30 22:09:23.433052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.433026 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.437575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.437555 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:23.437820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.437647 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:09:23.437925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.437739 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:23.438076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.437782 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:23.440390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.440366 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk"] Oct 30 22:09:23.538383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.538349 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.538383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.538390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlzb\" (UniqueName: \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.538879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.538853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.640687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.640653 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.640687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.640693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.640911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.640715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlzb\" (UniqueName: \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.641331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.641299 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.643337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.643313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.654535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.654513 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlzb\" (UniqueName: \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\") pod \"maas-api-revoke-sub-a99e4748-tzgmk\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:23.751704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:23.751623 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:24.060383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:24.060356 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk"] Oct 30 22:09:24.064786 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:09:24.064755 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a60d37_01d1_4a34_9323_3bed2efc638e.slice/crio-4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be WatchSource:0}: Error finding container 4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be: Status 404 returned error can't find the container with id 4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be Oct 30 22:09:24.733524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:24.733432 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" event={"ID":"64a60d37-01d1-4a34-9323-3bed2efc638e","Type":"ContainerStarted","Data":"7bcce5ddcc78215b8d4533a06ad65d13d638760aa5566f059c3c64baf80391df"} Oct 30 22:09:24.733524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:24.733503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" event={"ID":"64a60d37-01d1-4a34-9323-3bed2efc638e","Type":"ContainerStarted","Data":"4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be"} Oct 30 22:09:24.753397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:24.753345 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" podStartSLOduration=1.753329833 podStartE2EDuration="1.753329833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:24.751452922 +0000 UTC m=+1788.701407868" watchObservedRunningTime="2026-10-30 22:09:24.753329833 +0000 UTC m=+1788.703284779" Oct 30 22:09:25.028659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:25.028582 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc"] Oct 30 22:09:25.038328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:25.038293 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f7b094-tt6lc"] Oct 30 22:09:25.745643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:25.745603 5036 generic.go:356] "Generic (PLEG): container finished" podID="64a60d37-01d1-4a34-9323-3bed2efc638e" containerID="7bcce5ddcc78215b8d4533a06ad65d13d638760aa5566f059c3c64baf80391df" exitCode=0 Oct 30 22:09:25.746100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:25.745650 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" event={"ID":"64a60d37-01d1-4a34-9323-3bed2efc638e","Type":"ContainerDied","Data":"7bcce5ddcc78215b8d4533a06ad65d13d638760aa5566f059c3c64baf80391df"} Oct 30 22:09:26.191376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:26.191342 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0bd096-3797-43ba-a0b7-d16d64f310d3" path="/var/lib/kubelet/pods/fe0bd096-3797-43ba-a0b7-d16d64f310d3/volumes" Oct 30 22:09:26.988190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:26.988163 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:27.112066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.111922 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\") pod \"64a60d37-01d1-4a34-9323-3bed2efc638e\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " Oct 30 22:09:27.112066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.112064 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\") pod \"64a60d37-01d1-4a34-9323-3bed2efc638e\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " Oct 30 22:09:27.112356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.112086 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\" (UniqueName: \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\") pod \"64a60d37-01d1-4a34-9323-3bed2efc638e\" (UID: \"64a60d37-01d1-4a34-9323-3bed2efc638e\") " Oct 30 22:09:27.112356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.112202 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca" pod "64a60d37-01d1-4a34-9323-3bed2efc638e" (UID: "64a60d37-01d1-4a34-9323-3bed2efc638e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:27.112924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.112906 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.114538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.114512 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca" pod "64a60d37-01d1-4a34-9323-3bed2efc638e" (UID: "64a60d37-01d1-4a34-9323-3bed2efc638e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:27.114662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.114644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb" pod "64a60d37-01d1-4a34-9323-3bed2efc638e" (UID: "64a60d37-01d1-4a34-9323-3bed2efc638e"). InnerVolumeSpecName "kube-api-access-lhlzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:27.214869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.214836 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a60d37-01d1-4a34-9323-3bed2efc638e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.214869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.214866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlzb\" (UniqueName: \"kubernetes.io/projected/64a60d37-01d1-4a34-9323-3bed2efc638e-kube-api-access-lhlzb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:27.774196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.774113 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" Oct 30 22:09:27.774196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.774136 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk" event={"ID":"64a60d37-01d1-4a34-9323-3bed2efc638e","Type":"ContainerDied","Data":"4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be"} Oct 30 22:09:27.774196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:27.774184 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd3fe7f54e7bb952d5a5f647e6205925f67ded307b8ec20b7152ecdacb1a1be" Oct 30 22:09:35.230182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:35.230136 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:09:36.831085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:36.831043 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:09:36.886175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:36.886136 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:09:36.890921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:36.890897 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:09:42.132392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.132360 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k"] Oct 30 22:09:42.133200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.133180 5036 cpu_manager.go:414] "containerMap: removing container" podUID="64a60d37-01d1-4a34-9323-3bed2efc638e" containerName="revoke-keys" Oct 30 22:09:42.133200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.133200 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a60d37-01d1-4a34-9323-3bed2efc638e" containerName="revoke-keys" Oct 30 22:09:42.133378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.133371 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a60d37-01d1-4a34-9323-3bed2efc638e" containerName="revoke-keys" Oct 30 22:09:42.137598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.137578 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.141308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.141281 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:42.141490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.141473 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:42.141585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.141536 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:42.141903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.141878 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:09:42.147017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.146991 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k"] Oct 30 22:09:42.223432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.223397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgbz\" (UniqueName: \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.223711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.223683 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.223880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.223862 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.339663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.335084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.339663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.335464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgbz\" (UniqueName: \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.339663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.335518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.339663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.336659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.375361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.375321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.377288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.376107 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgbz\" (UniqueName: \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\") pod \"maas-api-revoke-sub-4a606d0d-98z7k\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.455942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.455861 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:42.775576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:42.775544 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k"] Oct 30 22:09:42.776767 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:09:42.776742 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58db2825_c2ac_4c18_98f2_815e691bf2e5.slice/crio-8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134 WatchSource:0}: Error finding container 8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134: Status 404 returned error can't find the container with id 8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134 Oct 30 22:09:43.026822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:43.026540 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" event={"ID":"58db2825-c2ac-4c18-98f2-815e691bf2e5","Type":"ContainerStarted","Data":"8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134"} Oct 30 22:09:44.039495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:44.039461 5036 generic.go:356] "Generic (PLEG): container finished" podID="58db2825-c2ac-4c18-98f2-815e691bf2e5" containerID="922cf120d2d0b89d8601da525f0cd204916807c2d9e9c6c72afb9cc34931b7f3" exitCode=0 Oct 30 22:09:44.039966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:44.039566 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" event={"ID":"58db2825-c2ac-4c18-98f2-815e691bf2e5","Type":"ContainerDied","Data":"922cf120d2d0b89d8601da525f0cd204916807c2d9e9c6c72afb9cc34931b7f3"} Oct 30 22:09:44.969320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:44.969282 5036 scope.go:122] "RemoveContainer" containerID="1ac73af5297a033f835f708d0cd973495b0ce9b693b31fa5a3cdf629e1c4cf29" Oct 30 22:09:44.993763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:44.993740 5036 scope.go:122] "RemoveContainer" containerID="3c588f4568cc3bb8ea33d3efa3838d0ad8d4de6e634842c1cd26ad57d2190e6f" Oct 30 22:09:45.006527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.006507 5036 scope.go:122] "RemoveContainer" containerID="a94efe1b61d3b188dd2d723bb9a1d2915edac2691dd0bb29f0bd2ea93cd58b90" Oct 30 22:09:45.017169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.017138 5036 scope.go:122] "RemoveContainer" containerID="af92070dd2eeb9c126ba92e85e85fdce0228e1c912c4aadbfc7017a96ac77f84" Oct 30 22:09:45.027873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.027852 5036 scope.go:122] "RemoveContainer" containerID="2f4a449518473bc7a6e60e5ea14ba9c6a0b3fe6527af7a69e4968d2a7a90b777" Oct 30 22:09:45.039407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.039384 5036 scope.go:122] "RemoveContainer" containerID="462800c268400baaf662a6a56f0d74f0ca53cdcf3320d2be94e9b68b6c9389bf" Oct 30 22:09:45.055935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.055915 5036 scope.go:122] "RemoveContainer" containerID="0110b48d40c09c417a311855c7bd7bacfeba9f78216dae543d92be718ae474f6" Oct 30 22:09:45.070920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.070892 5036 scope.go:122] "RemoveContainer" containerID="a254cf34dd8779f8723d162245e955c9c8921e89b9a63bc1a1ab1c8f97b51600" Oct 30 22:09:45.083248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.083219 5036 scope.go:122] "RemoveContainer" containerID="e847f8e4f9deaa932db67967347809821bb1eb55ac2107775dc4f5938f2c4681" Oct 30 22:09:45.095440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.095419 5036 scope.go:122] "RemoveContainer" containerID="d0c718a471991bb4fc289fe21923884195ac61dc4baaa22ba13c91137914ea78" Oct 30 22:09:45.109126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.109102 5036 scope.go:122] "RemoveContainer" containerID="c4fe98d4fb81b9a64309f6a9c00d5d52537804d004881e8801f9d4614c94fa4b" Oct 30 22:09:45.122162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.122142 5036 scope.go:122] "RemoveContainer" containerID="3a5e151edeb6a1a86c2f3a151982618e2e7af73e5462a379588567d0ed159fed" Oct 30 22:09:45.133306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.133278 5036 scope.go:122] "RemoveContainer" containerID="090d7c81f42aa572a0c0219441b85b20c84bff8cff7fd65709a3c8111f0c5bf4" Oct 30 22:09:45.291512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.291490 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:45.389177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.389138 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\") pod \"58db2825-c2ac-4c18-98f2-815e691bf2e5\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " Oct 30 22:09:45.389396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.389340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\") pod \"58db2825-c2ac-4c18-98f2-815e691bf2e5\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " Oct 30 22:09:45.389511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.389496 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\" (UniqueName: \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\") pod \"58db2825-c2ac-4c18-98f2-815e691bf2e5\" (UID: \"58db2825-c2ac-4c18-98f2-815e691bf2e5\") " Oct 30 22:09:45.389749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.389723 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca" pod "58db2825-c2ac-4c18-98f2-815e691bf2e5" (UID: "58db2825-c2ac-4c18-98f2-815e691bf2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:45.390137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.390118 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:45.392672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.392629 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca" pod "58db2825-c2ac-4c18-98f2-815e691bf2e5" (UID: "58db2825-c2ac-4c18-98f2-815e691bf2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:45.392854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.392816 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz" pod "58db2825-c2ac-4c18-98f2-815e691bf2e5" (UID: "58db2825-c2ac-4c18-98f2-815e691bf2e5"). InnerVolumeSpecName "kube-api-access-9cgbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:45.491963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.491889 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58db2825-c2ac-4c18-98f2-815e691bf2e5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:45.491963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:45.491931 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgbz\" (UniqueName: \"kubernetes.io/projected/58db2825-c2ac-4c18-98f2-815e691bf2e5-kube-api-access-9cgbz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.078478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:46.078451 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" Oct 30 22:09:46.078927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:46.078471 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k" event={"ID":"58db2825-c2ac-4c18-98f2-815e691bf2e5","Type":"ContainerDied","Data":"8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134"} Oct 30 22:09:46.078927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:46.078559 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8696d0072c4c9f5b5857e7ff0088eeba3f531f6808829b0695964fda9db1f134" Oct 30 22:09:48.100978 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:09:48.100940 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:09:50.262317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.262258 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerName="main" containerID="cri-o://caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183" gracePeriod=45 Oct 30 22:09:50.326205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.326159 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerName="main" probeResult="failure" output="Get \"https://10.132.1.2:8000/ready\": dial tcp 10.132.1.2:8000: connect: connection refused" Oct 30 22:09:50.671444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.671415 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:09:50.791842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.791809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792016 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792099 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:50.792203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792160 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\" (UniqueName: \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792216 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792328 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792389 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\") pod \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\" (UID: \"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72\") " Oct 30 22:09:50.792550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792518 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:50.792720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.792695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:50.793510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.793480 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.793634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.793516 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.793634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.793532 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.794711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.794683 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:50.794805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.794757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "kube-api-access-t4brc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:50.794805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.794765 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs" pod "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" (UID: "5c0b8b0d-ec77-4a40-afca-9a622c6c6b72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:50.908291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.905714 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.908291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.905766 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4brc\" (UniqueName: \"kubernetes.io/projected/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-kube-api-access-t4brc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.908291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:50.905783 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.129012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.128976 5036 generic.go:356] "Generic (PLEG): container finished" podID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerID="caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183" exitCode=0 Oct 30 22:09:51.129219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.129022 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" Oct 30 22:09:51.129219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.129054 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" event={"ID":"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72","Type":"ContainerDied","Data":"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183"} Oct 30 22:09:51.129219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.129109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv" event={"ID":"5c0b8b0d-ec77-4a40-afca-9a622c6c6b72","Type":"ContainerDied","Data":"fac6ec776c0fe0111004865a102bf05366002f031e15fec7ee0b22d99ed98d19"} Oct 30 22:09:51.129219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.129130 5036 scope.go:122] "RemoveContainer" containerID="caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183" Oct 30 22:09:51.147593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.147571 5036 scope.go:122] "RemoveContainer" containerID="caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183" Oct 30 22:09:51.147936 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:09:51.147913 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183\": container with ID starting with caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183 not found: ID does not exist" containerID="caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183" Oct 30 22:09:51.148032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.147943 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183"} err="failed to get container status \"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183\": rpc error: code = NotFound desc = could not find container \"caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183\": container with ID starting with caf2dff0756ac8337415ddbcae480fce4b7ade6abd15a94f434ed402b8def183 not found: ID does not exist" Oct 30 22:09:51.157186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.157163 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:09:51.163602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.163552 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-18ba2f/auth-test-model-18ba2f-kserve-6475786665-gd4jv"] Oct 30 22:09:51.866068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.866031 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerName="main" containerID="cri-o://b65e5a002e560f554d6ddc2af5cfe3f89c9da9696fbe5e0d238e684e7235323b" gracePeriod=45 Oct 30 22:09:51.869668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:51.869639 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerName="main" probeResult="failure" output="Get \"https://10.132.1.5:8000/ready\": dial tcp 10.132.1.5:8000: connect: connection refused" Oct 30 22:09:52.147569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.147538 5036 generic.go:356] "Generic (PLEG): container finished" podID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerID="b65e5a002e560f554d6ddc2af5cfe3f89c9da9696fbe5e0d238e684e7235323b" exitCode=0 Oct 30 22:09:52.147748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.147606 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" event={"ID":"37dec2d7-fc83-40ab-b66b-94272db2f376","Type":"ContainerDied","Data":"b65e5a002e560f554d6ddc2af5cfe3f89c9da9696fbe5e0d238e684e7235323b"} Oct 30 22:09:52.191033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.191004 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" path="/var/lib/kubelet/pods/5c0b8b0d-ec77-4a40-afca-9a622c6c6b72/volumes" Oct 30 22:09:52.195351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.195331 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:09:52.332488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332459 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.332687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332581 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.332687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332665 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\" (UniqueName: \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.332811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:52.332915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332889 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.333060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.332991 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.333060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\") pod \"37dec2d7-fc83-40ab-b66b-94272db2f376\" (UID: \"37dec2d7-fc83-40ab-b66b-94272db2f376\") " Oct 30 22:09:52.333188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333120 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:52.333324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333296 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:52.333739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333710 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.333739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333732 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.333927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.333746 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.335535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.335511 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:52.335709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.335691 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:52.335932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.335907 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s" pod "37dec2d7-fc83-40ab-b66b-94272db2f376" (UID: "37dec2d7-fc83-40ab-b66b-94272db2f376"). InnerVolumeSpecName "kube-api-access-jrt6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:52.435819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.435738 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/37dec2d7-fc83-40ab-b66b-94272db2f376-kube-api-access-jrt6s\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.435819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.435764 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dec2d7-fc83-40ab-b66b-94272db2f376-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.435819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.435776 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dec2d7-fc83-40ab-b66b-94272db2f376-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.666273 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.666239 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm"] Oct 30 22:09:52.667024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667007 5036 cpu_manager.go:414] "containerMap: removing container" podUID="58db2825-c2ac-4c18-98f2-815e691bf2e5" containerName="revoke-keys" Oct 30 22:09:52.667087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667039 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58db2825-c2ac-4c18-98f2-815e691bf2e5" containerName="revoke-keys" Oct 30 22:09:52.667087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667056 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerName="main" Oct 30 22:09:52.667087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667061 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerName="main" Oct 30 22:09:52.667182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667095 5036 cpu_manager.go:414] "containerMap: removing container" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerName="main" Oct 30 22:09:52.667182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667100 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerName="main" Oct 30 22:09:52.667252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667232 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="58db2825-c2ac-4c18-98f2-815e691bf2e5" containerName="revoke-keys" Oct 30 22:09:52.667252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667240 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0b8b0d-ec77-4a40-afca-9a622c6c6b72" containerName="main" Oct 30 22:09:52.667372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.667255 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" containerName="main" Oct 30 22:09:52.671393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.671374 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.674913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.674891 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:52.674913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.674901 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:52.675171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.674924 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:52.675246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.675206 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:09:52.679591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.679573 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm"] Oct 30 22:09:52.739571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.739491 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.739571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.739557 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.739776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.739696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4s6\" (UniqueName: \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.856174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.848666 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4s6\" (UniqueName: \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.856174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.849064 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.856174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.849115 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.866369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.866339 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.866854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.866454 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.872407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.870095 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4s6\" (UniqueName: \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\") pod \"maas-api-revoke-sub-afd21abd-86qpm\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:52.990815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:52.990740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:53.168719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.168648 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" Oct 30 22:09:53.168719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.168688 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9" event={"ID":"37dec2d7-fc83-40ab-b66b-94272db2f376","Type":"ContainerDied","Data":"571212b8698bc4f2bf57429fa81854c8970627fc760f9dec1ddfdd76aded52d1"} Oct 30 22:09:53.168977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.168741 5036 scope.go:122] "RemoveContainer" containerID="b65e5a002e560f554d6ddc2af5cfe3f89c9da9696fbe5e0d238e684e7235323b" Oct 30 22:09:53.195935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.195911 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:09:53.200670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.200646 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-38d6e2/auth-test-model-38d6e2-kserve-74bd9fc5cc-mmjr9"] Oct 30 22:09:53.288466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:53.288440 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm"] Oct 30 22:09:53.288635 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:09:53.288618 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf601b29d_3946_4486_af58_b3a9db116537.slice/crio-d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16 WatchSource:0}: Error finding container d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16: Status 404 returned error can't find the container with id d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16 Oct 30 22:09:54.190083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:54.190043 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37dec2d7-fc83-40ab-b66b-94272db2f376" path="/var/lib/kubelet/pods/37dec2d7-fc83-40ab-b66b-94272db2f376/volumes" Oct 30 22:09:54.190491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:54.190453 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" event={"ID":"f601b29d-3946-4486-af58-b3a9db116537","Type":"ContainerStarted","Data":"cc10714da1657fa03c0e5fde962018673604d12b7030391a79d01284b0f9cf77"} Oct 30 22:09:54.190533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:54.190501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" event={"ID":"f601b29d-3946-4486-af58-b3a9db116537","Type":"ContainerStarted","Data":"d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16"} Oct 30 22:09:54.201923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:54.201882 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" podStartSLOduration=2.201869348 podStartE2EDuration="2.201869348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:54.198773869 +0000 UTC m=+1818.148728816" watchObservedRunningTime="2026-10-30 22:09:54.201869348 +0000 UTC m=+1818.151824321" Oct 30 22:09:55.191191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:55.191160 5036 generic.go:356] "Generic (PLEG): container finished" podID="f601b29d-3946-4486-af58-b3a9db116537" containerID="cc10714da1657fa03c0e5fde962018673604d12b7030391a79d01284b0f9cf77" exitCode=0 Oct 30 22:09:55.191725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:55.191208 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" event={"ID":"f601b29d-3946-4486-af58-b3a9db116537","Type":"ContainerDied","Data":"cc10714da1657fa03c0e5fde962018673604d12b7030391a79d01284b0f9cf77"} Oct 30 22:09:56.828946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.828914 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:56.971396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.971302 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\" (UniqueName: \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\") pod \"f601b29d-3946-4486-af58-b3a9db116537\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " Oct 30 22:09:56.972885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.972858 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\") pod \"f601b29d-3946-4486-af58-b3a9db116537\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " Oct 30 22:09:56.973511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.973491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\") pod \"f601b29d-3946-4486-af58-b3a9db116537\" (UID: \"f601b29d-3946-4486-af58-b3a9db116537\") " Oct 30 22:09:56.973996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.973433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca" pod "f601b29d-3946-4486-af58-b3a9db116537" (UID: "f601b29d-3946-4486-af58-b3a9db116537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:56.977921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.977652 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f601b29d-3946-4486-af58-b3a9db116537-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:56.977921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.977826 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6" pod "f601b29d-3946-4486-af58-b3a9db116537" (UID: "f601b29d-3946-4486-af58-b3a9db116537"). InnerVolumeSpecName "kube-api-access-2j4s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:56.978054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:56.977946 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca" pod "f601b29d-3946-4486-af58-b3a9db116537" (UID: "f601b29d-3946-4486-af58-b3a9db116537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:57.093024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:57.092980 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4s6\" (UniqueName: \"kubernetes.io/projected/f601b29d-3946-4486-af58-b3a9db116537-kube-api-access-2j4s6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.093254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:57.093233 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f601b29d-3946-4486-af58-b3a9db116537-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.215827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:57.215797 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" Oct 30 22:09:57.216004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:57.215826 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm" event={"ID":"f601b29d-3946-4486-af58-b3a9db116537","Type":"ContainerDied","Data":"d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16"} Oct 30 22:09:57.216004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:57.215877 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d354644e127eecebbceb36a4a45dd3f90c593e176985f67a0646626c684bfc16" Oct 30 22:09:58.925223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.925171 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:09:58.927095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.927065 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f601b29d-3946-4486-af58-b3a9db116537" containerName="revoke-keys" Oct 30 22:09:58.927303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.927283 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f601b29d-3946-4486-af58-b3a9db116537" containerName="revoke-keys" Oct 30 22:09:58.927650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.927630 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f601b29d-3946-4486-af58-b3a9db116537" containerName="revoke-keys" Oct 30 22:09:58.932723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.932692 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:58.947794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.947767 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:09:58.950222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:58.950180 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1a5bdf-openshift-default-dockercfg-s5q47\"" Oct 30 22:09:59.053084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053044 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053100 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053216 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053342 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053431 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrv5z\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053567 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.053720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.053699 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155259 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155524 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155648 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrv5z\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155830 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.155931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155911 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.156183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.155972 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.156247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.156226 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.156376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.156332 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.156552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.156530 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.158565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.158544 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.158833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.158813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.198337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.195977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrv5z\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.198337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.196716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\") pod \"e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.292389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.290560 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:09:59.353292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.352307 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84"] Oct 30 22:09:59.378289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.376744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.391292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.381309 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:59.415288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.406869 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:59.437289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.431607 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:09:59.437289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.431715 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:59.437289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.432190 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84"] Oct 30 22:09:59.586742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.586703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.586922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.586901 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.586991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.586965 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtkn\" (UniqueName: \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.688699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.688655 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.688898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.688722 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtkn\" (UniqueName: \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.688980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.688959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.689662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.689643 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.691229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.691211 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.700956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.700929 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtkn\" (UniqueName: \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\") pod \"maas-api-revoke-sub-cdbeec18-ppm84\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:09:59.816238 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:09:59.816205 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbdf196_4f1a_4542_80d9_b83dafc230b1.slice/crio-fc8c72ba52da2652135306506d9f5991a66d7ddd7c829d0df1da026701962a93 WatchSource:0}: Error finding container fc8c72ba52da2652135306506d9f5991a66d7ddd7c829d0df1da026701962a93: Status 404 returned error can't find the container with id fc8c72ba52da2652135306506d9f5991a66d7ddd7c829d0df1da026701962a93 Oct 30 22:09:59.818895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.818866 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:09:59.819081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.819048 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:59.819171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.819155 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:59.819211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.819198 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:59.828757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:09:59.828731 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:10:00.138320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:00.137436 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84"] Oct 30 22:10:00.372907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:00.372867 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerStarted","Data":"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404"} Oct 30 22:10:00.373088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:00.372935 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerStarted","Data":"fc8c72ba52da2652135306506d9f5991a66d7ddd7c829d0df1da026701962a93"} Oct 30 22:10:00.374600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:00.374574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" event={"ID":"d22204fc-bb5b-4eec-bb15-4e83ae18078a","Type":"ContainerStarted","Data":"c61da8b4cf7a5037be2aa981f5de635cfa27b05d8a6186696e9fd9cc6e11082e"} Oct 30 22:10:00.398661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:00.398550 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podStartSLOduration=2.398533058 podStartE2EDuration="2.398533058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:00.396709161 +0000 UTC m=+1824.346664107" watchObservedRunningTime="2026-10-30 22:10:00.398533058 +0000 UTC m=+1824.348488005" Oct 30 22:10:01.295190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.295160 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:01.301911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.301884 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:01.386057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.386023 5036 generic.go:356] "Generic (PLEG): container finished" podID="d22204fc-bb5b-4eec-bb15-4e83ae18078a" containerID="5966ad500e7bd42f28ca6515fa2449b755b4d3e5eda29da8e69946c0ab0d3596" exitCode=0 Oct 30 22:10:01.386225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.386111 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" event={"ID":"d22204fc-bb5b-4eec-bb15-4e83ae18078a","Type":"ContainerDied","Data":"5966ad500e7bd42f28ca6515fa2449b755b4d3e5eda29da8e69946c0ab0d3596"} Oct 30 22:10:01.386539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.386518 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:01.387555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:01.387537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:02.586795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.586766 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:10:02.681666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.681590 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\") pod \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " Oct 30 22:10:02.681666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.681626 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\") pod \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " Oct 30 22:10:02.681950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.681926 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\" (UniqueName: \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\") pod \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\" (UID: \"d22204fc-bb5b-4eec-bb15-4e83ae18078a\") " Oct 30 22:10:02.682180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.682003 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca" pod "d22204fc-bb5b-4eec-bb15-4e83ae18078a" (UID: "d22204fc-bb5b-4eec-bb15-4e83ae18078a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:02.682784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.682763 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.684215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.684195 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn" pod "d22204fc-bb5b-4eec-bb15-4e83ae18078a" (UID: "d22204fc-bb5b-4eec-bb15-4e83ae18078a"). InnerVolumeSpecName "kube-api-access-ndtkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.684623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.684604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca" pod "d22204fc-bb5b-4eec-bb15-4e83ae18078a" (UID: "d22204fc-bb5b-4eec-bb15-4e83ae18078a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.784254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.784218 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22204fc-bb5b-4eec-bb15-4e83ae18078a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.784254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:02.784249 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtkn\" (UniqueName: \"kubernetes.io/projected/d22204fc-bb5b-4eec-bb15-4e83ae18078a-kube-api-access-ndtkn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.411362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:03.411327 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" Oct 30 22:10:03.411556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:03.411466 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84" event={"ID":"d22204fc-bb5b-4eec-bb15-4e83ae18078a","Type":"ContainerDied","Data":"c61da8b4cf7a5037be2aa981f5de635cfa27b05d8a6186696e9fd9cc6e11082e"} Oct 30 22:10:03.411556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:03.411505 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61da8b4cf7a5037be2aa981f5de635cfa27b05d8a6186696e9fd9cc6e11082e" Oct 30 22:10:04.624286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.610334 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:10:04.624286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.611750 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d22204fc-bb5b-4eec-bb15-4e83ae18078a" containerName="revoke-keys" Oct 30 22:10:04.624286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.611776 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22204fc-bb5b-4eec-bb15-4e83ae18078a" containerName="revoke-keys" Oct 30 22:10:04.624286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.612027 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22204fc-bb5b-4eec-bb15-4e83ae18078a" containerName="revoke-keys" Oct 30 22:10:04.653292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.637347 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:10:04.653292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.637447 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.653292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.642850 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0a2577-openshift-default-dockercfg-6pss4\"" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782384 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782511 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782541 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sl6\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782700 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.782935 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.783016 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.794288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.783107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886078 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886137 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sl6\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886203 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886389 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886551 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.886799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.887370 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.887610 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.888420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.888155 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.895292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.892742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.895292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.892981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.903521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.899849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.903589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.903533 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.907286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.907075 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sl6\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.910144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.907504 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:04.957173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:04.956588 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:05.311165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.311129 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:10:05.318321 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:05.316563 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085c6a13_c475_4a63_a664_9d2bfdd584b5.slice/crio-537c068fa98f265ae4f4c81a8869708f12bd2e7d5d581a8790ff3e73b0767083 WatchSource:0}: Error finding container 537c068fa98f265ae4f4c81a8869708f12bd2e7d5d581a8790ff3e73b0767083: Status 404 returned error can't find the container with id 537c068fa98f265ae4f4c81a8869708f12bd2e7d5d581a8790ff3e73b0767083 Oct 30 22:10:05.322615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.322571 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:05.323303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.323258 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:05.323608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.323532 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:05.633794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.633759 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" event={"ID":"085c6a13-c475-4a63-a664-9d2bfdd584b5","Type":"ContainerStarted","Data":"9d8e960d4b1dd438b4c6b58c1485a7f9c51b61747dd72a5d4be570da10563387"} Oct 30 22:10:05.634295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.633827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" event={"ID":"085c6a13-c475-4a63-a664-9d2bfdd584b5","Type":"ContainerStarted","Data":"537c068fa98f265ae4f4c81a8869708f12bd2e7d5d581a8790ff3e73b0767083"} Oct 30 22:10:05.958045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:05.957963 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:06.959611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:06.959579 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:06.960338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:06.959632 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:07.957860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:07.957823 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:07.958038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:07.957897 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:08.960386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:08.957680 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:08.960386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:08.957750 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:09.663065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.662922 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podStartSLOduration=5.662896703 podStartE2EDuration="5.662896703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:05.664509605 +0000 UTC m=+1829.614464553" watchObservedRunningTime="2026-10-30 22:10:09.662896703 +0000 UTC m=+1833.612851657" Oct 30 22:10:09.666621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.666589 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x"] Oct 30 22:10:09.690623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.690590 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.690932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.690633 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x"] Oct 30 22:10:09.698738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.698718 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:09.698861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.698735 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:09.699578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.699559 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:09.699668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.699611 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:10:09.760180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.760146 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtsgk\" (UniqueName: \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.760484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.760454 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.760615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.760591 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.861980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.861943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.862183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.862057 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsgk\" (UniqueName: \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.862320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.862301 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.862925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.862898 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.864714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.864691 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.874452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.874426 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtsgk\" (UniqueName: \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\") pod \"maas-api-revoke-sub-87235e6d-ztd5x\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:09.957497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.957399 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:09.957497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:09.957447 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:10.041555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:10.041518 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:10.383468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:10.383368 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x"] Oct 30 22:10:10.704141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:10.704109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" event={"ID":"4c17be6f-6ca1-4625-a31a-524c21682cfb","Type":"ContainerStarted","Data":"b1276ab354c7b0c4d050e753ec158f25fcf186cf74be107ec2cd885982910a43"} Oct 30 22:10:10.964344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:10.960692 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:10.964344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:10.960758 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:11.729755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:11.729713 5036 generic.go:356] "Generic (PLEG): container finished" podID="4c17be6f-6ca1-4625-a31a-524c21682cfb" containerID="872996e363dae51c9637857e0f07f60fd740f9902a0acaa1bad0deedbd21b422" exitCode=0 Oct 30 22:10:11.730326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:11.729766 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" event={"ID":"4c17be6f-6ca1-4625-a31a-524c21682cfb","Type":"ContainerDied","Data":"872996e363dae51c9637857e0f07f60fd740f9902a0acaa1bad0deedbd21b422"} Oct 30 22:10:11.957637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:11.957601 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 22:10:11.957871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:11.957696 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 22:10:12.933588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:12.933560 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:12.964070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:12.964039 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:12.964461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:12.964437 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:12.965609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:12.965589 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:10:13.035507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.035427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\") pod \"4c17be6f-6ca1-4625-a31a-524c21682cfb\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " Oct 30 22:10:13.035759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.035736 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\" (UniqueName: \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\") pod \"4c17be6f-6ca1-4625-a31a-524c21682cfb\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " Oct 30 22:10:13.036037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.036014 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\") pod \"4c17be6f-6ca1-4625-a31a-524c21682cfb\" (UID: \"4c17be6f-6ca1-4625-a31a-524c21682cfb\") " Oct 30 22:10:13.037091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.037062 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca" pod "4c17be6f-6ca1-4625-a31a-524c21682cfb" (UID: "4c17be6f-6ca1-4625-a31a-524c21682cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:13.042644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.040872 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk" pod "4c17be6f-6ca1-4625-a31a-524c21682cfb" (UID: "4c17be6f-6ca1-4625-a31a-524c21682cfb"). InnerVolumeSpecName "kube-api-access-mtsgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:13.055757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.051219 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtsgk\" (UniqueName: \"kubernetes.io/projected/4c17be6f-6ca1-4625-a31a-524c21682cfb-kube-api-access-mtsgk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:13.055757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.051245 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:13.069763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.069719 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca" pod "4c17be6f-6ca1-4625-a31a-524c21682cfb" (UID: "4c17be6f-6ca1-4625-a31a-524c21682cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:13.178442 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.175549 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c17be6f-6ca1-4625-a31a-524c21682cfb-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:13.752251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.752215 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" Oct 30 22:10:13.752486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.752285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x" event={"ID":"4c17be6f-6ca1-4625-a31a-524c21682cfb","Type":"ContainerDied","Data":"b1276ab354c7b0c4d050e753ec158f25fcf186cf74be107ec2cd885982910a43"} Oct 30 22:10:13.752486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:13.752345 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1276ab354c7b0c4d050e753ec158f25fcf186cf74be107ec2cd885982910a43" Oct 30 22:10:18.101832 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:10:18.101745 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:10:26.438549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.438516 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:10:26.439370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.439340 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" containerID="cri-o://a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4" gracePeriod=30 Oct 30 22:10:26.628033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.627996 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:10:26.628878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.628857 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4c17be6f-6ca1-4625-a31a-524c21682cfb" containerName="revoke-keys" Oct 30 22:10:26.628878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.628877 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c17be6f-6ca1-4625-a31a-524c21682cfb" containerName="revoke-keys" Oct 30 22:10:26.629063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.629019 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c17be6f-6ca1-4625-a31a-524c21682cfb" containerName="revoke-keys" Oct 30 22:10:26.633084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.633063 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.639403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.639378 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-26a128\"" Oct 30 22:10:26.644504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.644480 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:10:26.671056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.671022 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.671213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.671184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8zlf\" (UniqueName: \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.814685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.773636 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.899508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.825337 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.899508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.890626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8zlf\" (UniqueName: \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.950522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.950405 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8zlf\" (UniqueName: \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\") pod \"payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:26.965026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:26.964740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:27.429893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:27.429865 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:10:27.430705 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:27.430675 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8d351a_b512_45eb_b0be_a0755fb9816b.slice/crio-b9549a287efcc9016ba2803f75f88fac0c4bec49a573c0e3b173b1086edadd29 WatchSource:0}: Error finding container b9549a287efcc9016ba2803f75f88fac0c4bec49a573c0e3b173b1086edadd29: Status 404 returned error can't find the container with id b9549a287efcc9016ba2803f75f88fac0c4bec49a573c0e3b173b1086edadd29 Oct 30 22:10:28.077524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:28.077484 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" event={"ID":"2e8d351a-b512-45eb-b0be-a0755fb9816b","Type":"ContainerStarted","Data":"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801"} Oct 30 22:10:28.078038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:28.077583 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" event={"ID":"2e8d351a-b512-45eb-b0be-a0755fb9816b","Type":"ContainerStarted","Data":"b9549a287efcc9016ba2803f75f88fac0c4bec49a573c0e3b173b1086edadd29"} Oct 30 22:10:28.078038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:28.077612 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:28.109704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:28.109655 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" podStartSLOduration=2.1096409879999998 podStartE2EDuration="2.109640988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:28.104039049 +0000 UTC m=+1852.053993999" watchObservedRunningTime="2026-10-30 22:10:28.109640988 +0000 UTC m=+1852.059595933" Oct 30 22:10:31.990212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:31.990177 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:10:32.079470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.079432 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.079757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.079732 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:32.080129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.079830 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.080594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.080318 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.080925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.080904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.080998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.080957 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.080998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.080992 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.081104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.081346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081325 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.081346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081334 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:32.081453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081431 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\") pod \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\" (UID: \"b0abe737-9a76-4ee3-b0c0-166e4297b30b\") " Oct 30 22:10:32.081609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081588 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:32.082109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.081880 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:32.082544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.082290 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.082544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.082315 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.082544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.082329 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.082544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.082342 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.082867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.082834 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:32.084188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.084153 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:32.084485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.084455 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "kube-api-access-zx5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:32.085151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.085103 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:32.085978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.085946 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo" pod "b0abe737-9a76-4ee3-b0c0-166e4297b30b" (UID: "b0abe737-9a76-4ee3-b0c0-166e4297b30b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:32.183807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.183716 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.183807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.183752 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.183807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.183767 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.183807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.183781 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx5qq\" (UniqueName: \"kubernetes.io/projected/b0abe737-9a76-4ee3-b0c0-166e4297b30b-kube-api-access-zx5qq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.183807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.183795 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe737-9a76-4ee3-b0c0-166e4297b30b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.204498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.204460 5036 generic.go:356] "Generic (PLEG): container finished" podID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerID="a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4" exitCode=0 Oct 30 22:10:32.204642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.204502 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" Oct 30 22:10:32.204642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.204535 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" event={"ID":"b0abe737-9a76-4ee3-b0c0-166e4297b30b","Type":"ContainerDied","Data":"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4"} Oct 30 22:10:32.204642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.204584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw" event={"ID":"b0abe737-9a76-4ee3-b0c0-166e4297b30b","Type":"ContainerDied","Data":"254a9e1b7713c61dff1bb867c6a2e6761ac06ca52be4c2a5796b9184596656b6"} Oct 30 22:10:32.204642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.204615 5036 scope.go:122] "RemoveContainer" containerID="a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4" Oct 30 22:10:32.251373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.251350 5036 scope.go:122] "RemoveContainer" containerID="a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4" Oct 30 22:10:32.251736 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:10:32.251713 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4\": container with ID starting with a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4 not found: ID does not exist" containerID="a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4" Oct 30 22:10:32.251798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.251745 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4"} err="failed to get container status \"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4\": rpc error: code = NotFound desc = could not find container \"a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4\": container with ID starting with a5b4735a97f1085449eb1e330cab89831634e39f38645dbfa1733390502807e4 not found: ID does not exist" Oct 30 22:10:32.262471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.262443 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:10:32.268789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:32.268768 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-2mkcw"] Oct 30 22:10:34.202414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.202383 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" path="/var/lib/kubelet/pods/b0abe737-9a76-4ee3-b0c0-166e4297b30b/volumes" Oct 30 22:10:34.252544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.252452 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258"] Oct 30 22:10:34.254009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.253981 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" Oct 30 22:10:34.254009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.254008 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" Oct 30 22:10:34.254281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.254254 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0abe737-9a76-4ee3-b0c0-166e4297b30b" containerName="istio-proxy" Oct 30 22:10:34.260620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.260512 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258"] Oct 30 22:10:34.260798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.260600 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.266008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.265985 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:34.269079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.268556 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:10:34.269079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.268609 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:34.269079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.268628 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:34.361187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.361144 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.361739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.361716 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l25j\" (UniqueName: \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.363297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.362083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.464477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.464370 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l25j\" (UniqueName: \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.464669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.464508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.464669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.464581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.465168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.465144 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.467095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.467077 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.476552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.476526 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l25j\" (UniqueName: \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\") pod \"maas-api-revoke-sub-083f264c-ps258\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.594761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.594723 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:34.890465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.890430 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:10:34.904544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.903758 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:34.956293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.911703 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:10:34.956651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:34.913496 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-28b4bb-openshift-default-dockercfg-bdj78\"" Oct 30 22:10:35.003083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.003039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.003528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.003479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.003865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.003846 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.004056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.004039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.004326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.004310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.004509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.004491 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.004721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.004702 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5986\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.004885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.004870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.005022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.005007 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.139744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.139788 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.139845 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5986\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.139878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.139904 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.140094 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.140180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.144168 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.144249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.149718 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.157617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.158801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.158360 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.271562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.161881 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.271562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.162145 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.340256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.323546 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5986\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.432521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.430476 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.432521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.431925 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.460756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.457461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:35.506776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.506744 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258"] Oct 30 22:10:35.661795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:35.659760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:36.304964 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:36.304932 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca4cc46_0c3b_4ac7_854c_6563870206b0.slice/crio-2507a1d0f92e5a485bf46f66d20c28621f1ab88811b9934d5274df090130fc8e WatchSource:0}: Error finding container 2507a1d0f92e5a485bf46f66d20c28621f1ab88811b9934d5274df090130fc8e: Status 404 returned error can't find the container with id 2507a1d0f92e5a485bf46f66d20c28621f1ab88811b9934d5274df090130fc8e Oct 30 22:10:36.306027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.305912 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:10:36.309406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.308499 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:36.309406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.308591 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:36.309406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.308634 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:36.541289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.533402 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" event={"ID":"3e1bcf3c-1792-44e6-a20d-784c343eaa03","Type":"ContainerStarted","Data":"3c00623eada62a5dafa6a89438848d6b1bf759960ad86e3010767a6b80ce05cd"} Oct 30 22:10:36.541289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.533469 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" event={"ID":"3e1bcf3c-1792-44e6-a20d-784c343eaa03","Type":"ContainerStarted","Data":"43901f1c6c1695e4cda4b6e4f3a66afc94088f87479a48ac5068ed2b84c26f47"} Oct 30 22:10:36.556430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.551536 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerStarted","Data":"2507a1d0f92e5a485bf46f66d20c28621f1ab88811b9934d5274df090130fc8e"} Oct 30 22:10:36.566571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.566505 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" podStartSLOduration=2.566490991 podStartE2EDuration="2.566490991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:36.560576385 +0000 UTC m=+1860.510531348" watchObservedRunningTime="2026-10-30 22:10:36.566490991 +0000 UTC m=+1860.516445934" Oct 30 22:10:36.703579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.703542 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:10:36.708520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.708497 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:36.732579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.732396 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b9d868\"" Oct 30 22:10:36.765652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.765623 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:10:36.780770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.780742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:36.781083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.781067 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5f56\" (UniqueName: \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:36.797223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.797189 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:10:36.806736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.806654 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:10:36.807014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.806757 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:36.836304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.836131 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b9d868-dockercfg-rrw4h\"" Oct 30 22:10:36.903291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.902394 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrxf\" (UniqueName: \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:36.903291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.902496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:36.903291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.902612 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:36.903291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.902744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5f56\" (UniqueName: \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:36.928298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:36.907777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:37.044293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.034771 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrxf\" (UniqueName: \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:37.044293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.034903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:37.044293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.036145 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:37.294564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.294506 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5f56\" (UniqueName: \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\") pod \"payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:37.296373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.296287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrxf\" (UniqueName: \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\") pod \"payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:37.374816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.374783 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:37.571873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.571601 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:37.572051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.571920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerStarted","Data":"4d14ca2a54c8528d009e7cd8975093a70a298de7052ffbc858b8a70a9a63c368"} Oct 30 22:10:37.585044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.585002 5036 generic.go:356] "Generic (PLEG): container finished" podID="3e1bcf3c-1792-44e6-a20d-784c343eaa03" containerID="3c00623eada62a5dafa6a89438848d6b1bf759960ad86e3010767a6b80ce05cd" exitCode=0 Oct 30 22:10:37.585195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.585067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" event={"ID":"3e1bcf3c-1792-44e6-a20d-784c343eaa03","Type":"ContainerDied","Data":"3c00623eada62a5dafa6a89438848d6b1bf759960ad86e3010767a6b80ce05cd"} Oct 30 22:10:37.597778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.597680 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podStartSLOduration=3.5976633700000002 podStartE2EDuration="3.59766337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:37.5946471 +0000 UTC m=+1861.544602052" watchObservedRunningTime="2026-10-30 22:10:37.59766337 +0000 UTC m=+1861.547618313" Oct 30 22:10:37.660843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.660766 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:37.669682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.669658 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:37.796682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.796582 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:10:37.892979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:37.892942 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:10:37.903007 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:37.902971 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52e5962_6fe0_4f8b_9a72_db5dc01c6a21.slice/crio-9a358b9f6b4932fc56f0a830489bba5e7f3cf935010c2c3e049a496f7be07403 WatchSource:0}: Error finding container 9a358b9f6b4932fc56f0a830489bba5e7f3cf935010c2c3e049a496f7be07403: Status 404 returned error can't find the container with id 9a358b9f6b4932fc56f0a830489bba5e7f3cf935010c2c3e049a496f7be07403 Oct 30 22:10:38.601128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.601083 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" event={"ID":"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21","Type":"ContainerStarted","Data":"aef199335db493a11ae4c78f9987c13086b9bab1aa32bd3c6b4eaf6d7b6c66fd"} Oct 30 22:10:38.601737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.601166 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:38.601737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.601195 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" event={"ID":"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21","Type":"ContainerStarted","Data":"9a358b9f6b4932fc56f0a830489bba5e7f3cf935010c2c3e049a496f7be07403"} Oct 30 22:10:38.603200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.603168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" event={"ID":"8f376211-2a5d-475b-adfa-394b499e2f52","Type":"ContainerStarted","Data":"c877e9af1086d892790d8415382a7eae963d937798891bdd428eca55fde4f52d"} Oct 30 22:10:38.603452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.603222 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" event={"ID":"8f376211-2a5d-475b-adfa-394b499e2f52","Type":"ContainerStarted","Data":"1e3c7d191ffb9574b9b0b68c6d0b6aab92bbff74ce1a0fb97401fa25076ba86e"} Oct 30 22:10:38.603452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.603274 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:38.603593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.603558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:38.605425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.605404 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:10:38.624783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.624730 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" podStartSLOduration=2.62471098 podStartE2EDuration="2.62471098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:38.618202964 +0000 UTC m=+1862.568157910" watchObservedRunningTime="2026-10-30 22:10:38.62471098 +0000 UTC m=+1862.574665926" Oct 30 22:10:38.644116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.644060 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" podStartSLOduration=2.644042265 podStartE2EDuration="2.644042265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:38.637050423 +0000 UTC m=+1862.587005370" watchObservedRunningTime="2026-10-30 22:10:38.644042265 +0000 UTC m=+1862.593997212" Oct 30 22:10:38.994290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:38.977490 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:39.104527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.104477 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:10:39.116667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.116636 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:10:39.116984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.116916 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" containerID="cri-o://ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3" gracePeriod=30 Oct 30 22:10:39.138229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.137679 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\" (UniqueName: \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\") pod \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " Oct 30 22:10:39.138229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.137756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\") pod \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " Oct 30 22:10:39.138229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.137802 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\") pod \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\" (UID: \"3e1bcf3c-1792-44e6-a20d-784c343eaa03\") " Oct 30 22:10:39.139643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.139340 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca" pod "3e1bcf3c-1792-44e6-a20d-784c343eaa03" (UID: "3e1bcf3c-1792-44e6-a20d-784c343eaa03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:39.147632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.147595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j" pod "3e1bcf3c-1792-44e6-a20d-784c343eaa03" (UID: "3e1bcf3c-1792-44e6-a20d-784c343eaa03"). InnerVolumeSpecName "kube-api-access-4l25j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:39.152160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.151731 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca" pod "3e1bcf3c-1792-44e6-a20d-784c343eaa03" (UID: "3e1bcf3c-1792-44e6-a20d-784c343eaa03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:39.240629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.240473 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:39.240629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.240505 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1bcf3c-1792-44e6-a20d-784c343eaa03-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:39.240629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.240522 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l25j\" (UniqueName: \"kubernetes.io/projected/3e1bcf3c-1792-44e6-a20d-784c343eaa03-kube-api-access-4l25j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:39.564460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.564333 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:10:39.564774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.564644 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" containerID="cri-o://ad54f10af3fa220ed666a32f5de071f75fc43f6bfd6b2f5252ad1764dee2e345" gracePeriod=30 Oct 30 22:10:39.573087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.573058 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 22:10:39.573279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.573241 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" containerID="cri-o://33bb390a5dcea96ca16bc13a527fd61c7c2791c63fcd2a3173e6d3f882ea97f2" gracePeriod=30 Oct 30 22:10:39.656443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.656394 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" event={"ID":"3e1bcf3c-1792-44e6-a20d-784c343eaa03","Type":"ContainerDied","Data":"43901f1c6c1695e4cda4b6e4f3a66afc94088f87479a48ac5068ed2b84c26f47"} Oct 30 22:10:39.656938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.656466 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43901f1c6c1695e4cda4b6e4f3a66afc94088f87479a48ac5068ed2b84c26f47" Oct 30 22:10:39.656938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:39.656630 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258" Oct 30 22:10:41.364725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.364685 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:10:41.366088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.366062 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3e1bcf3c-1792-44e6-a20d-784c343eaa03" containerName="revoke-keys" Oct 30 22:10:41.366173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.366091 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1bcf3c-1792-44e6-a20d-784c343eaa03" containerName="revoke-keys" Oct 30 22:10:41.366460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.366440 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1bcf3c-1792-44e6-a20d-784c343eaa03" containerName="revoke-keys" Oct 30 22:10:41.373222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.373199 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.379094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.379066 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:10:41.382298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.382011 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-28b4bb-dockercfg-27ngt\"" Oct 30 22:10:41.383254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.383220 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-28b4bb\"" Oct 30 22:10:41.484290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.481734 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29gp\" (UniqueName: \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.484290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.481869 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.624338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.613539 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z29gp\" (UniqueName: \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.624338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.613795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.624338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.615494 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.776255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.776221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29gp\" (UniqueName: \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\") pod \"payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:41.993972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:41.993844 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:42.321772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:42.321740 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:10:42.323257 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:42.323233 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80a64c9_ef5d_4057_9b42_0d5050fb7419.slice/crio-74df7fe85640f19ff9039710be0dde709ab3444def871c49ea6550040212e4e5 WatchSource:0}: Error finding container 74df7fe85640f19ff9039710be0dde709ab3444def871c49ea6550040212e4e5: Status 404 returned error can't find the container with id 74df7fe85640f19ff9039710be0dde709ab3444def871c49ea6550040212e4e5 Oct 30 22:10:42.796153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:42.796116 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" event={"ID":"a80a64c9-ef5d-4057-9b42-0d5050fb7419","Type":"ContainerStarted","Data":"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78"} Oct 30 22:10:42.796680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:42.796180 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" event={"ID":"a80a64c9-ef5d-4057-9b42-0d5050fb7419","Type":"ContainerStarted","Data":"74df7fe85640f19ff9039710be0dde709ab3444def871c49ea6550040212e4e5"} Oct 30 22:10:42.796680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:42.796208 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:42.856295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:42.853793 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" podStartSLOduration=1.853773233 podStartE2EDuration="1.853773233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:42.85352736 +0000 UTC m=+1866.803482308" watchObservedRunningTime="2026-10-30 22:10:42.853773233 +0000 UTC m=+1866.803728181" Oct 30 22:10:43.807283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.807175 5036 generic.go:356] "Generic (PLEG): container finished" podID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerID="3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404" exitCode=0 Oct 30 22:10:43.807283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.807242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerDied","Data":"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404"} Oct 30 22:10:43.807893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.807706 5036 scope.go:122] "RemoveContainer" containerID="3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404" Oct 30 22:10:43.809609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.809579 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:43.809713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.809646 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:43.809713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:43.809697 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:44.468539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.468444 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2"] Oct 30 22:10:44.499193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.499166 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2"] Oct 30 22:10:44.499452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.499362 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.502524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.502501 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:44.502966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.502945 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:44.503060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.502964 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:44.503060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.503006 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:10:44.525639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.525614 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:10:44.620673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.620638 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.620846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.620686 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.620846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.620834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6klz\" (UniqueName: \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.721741 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.721850 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.721909 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722088 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722134 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722193 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\") pod \"5b177cd1-0be4-49f1-b550-6d896570a85b\" (UID: \"5b177cd1-0be4-49f1-b550-6d896570a85b\") " Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.722864 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.723349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.723463 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6klz\" (UniqueName: \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.723931 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.723237 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.724232 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.724759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.724805 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:44.726334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.725029 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:44.736032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.735957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.736534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.736472 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:44.740828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.740787 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "kube-api-access-dc4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:44.741055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.741008 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:44.741416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.741381 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo" pod "5b177cd1-0be4-49f1-b550-6d896570a85b" (UID: "5b177cd1-0be4-49f1-b550-6d896570a85b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:44.750922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.750758 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6klz\" (UniqueName: \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\") pod \"maas-api-revoke-sub-1697d7ca-vktk2\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829734 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829773 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829788 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829804 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829818 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b177cd1-0be4-49f1-b550-6d896570a85b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829832 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829847 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4ch\" (UniqueName: \"kubernetes.io/projected/5b177cd1-0be4-49f1-b550-6d896570a85b-kube-api-access-dc4ch\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829863 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.841301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.829877 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b177cd1-0be4-49f1-b550-6d896570a85b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:44.868096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.867668 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:44.914347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.913953 5036 generic.go:356] "Generic (PLEG): container finished" podID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerID="33bb390a5dcea96ca16bc13a527fd61c7c2791c63fcd2a3173e6d3f882ea97f2" exitCode=0 Oct 30 22:10:44.914347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.914237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" event={"ID":"2445cf25-888e-4836-971e-ac2ecd90eea5","Type":"ContainerDied","Data":"33bb390a5dcea96ca16bc13a527fd61c7c2791c63fcd2a3173e6d3f882ea97f2"} Oct 30 22:10:44.924945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.922135 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerStarted","Data":"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd"} Oct 30 22:10:44.941971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.941940 5036 generic.go:356] "Generic (PLEG): container finished" podID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerID="ad54f10af3fa220ed666a32f5de071f75fc43f6bfd6b2f5252ad1764dee2e345" exitCode=0 Oct 30 22:10:44.942331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.942256 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" event={"ID":"59f2deb7-9007-4a20-9931-319aad1e0c2d","Type":"ContainerDied","Data":"ad54f10af3fa220ed666a32f5de071f75fc43f6bfd6b2f5252ad1764dee2e345"} Oct 30 22:10:44.945400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.945371 5036 generic.go:356] "Generic (PLEG): container finished" podID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerID="ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3" exitCode=0 Oct 30 22:10:44.945530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.945449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" event={"ID":"5b177cd1-0be4-49f1-b550-6d896570a85b","Type":"ContainerDied","Data":"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3"} Oct 30 22:10:44.945530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.945453 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" Oct 30 22:10:44.945530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.945495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp" event={"ID":"5b177cd1-0be4-49f1-b550-6d896570a85b","Type":"ContainerDied","Data":"bdd7957bf828dcd5e53b7880a9d53c52557781fb5e2215ddb87586801c820b77"} Oct 30 22:10:44.945530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:44.945523 5036 scope.go:122] "RemoveContainer" containerID="ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3" Oct 30 22:10:45.017498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.017073 5036 scope.go:122] "RemoveContainer" containerID="ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3" Oct 30 22:10:45.019642 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:10:45.019490 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3\": container with ID starting with ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3 not found: ID does not exist" containerID="ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3" Oct 30 22:10:45.019642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.019558 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3"} err="failed to get container status \"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3\": rpc error: code = NotFound desc = could not find container \"ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3\": container with ID starting with ceba9e65ce7b10bebe0fbb210526f038409e4421fe12608a329c3ca3b0b77af3 not found: ID does not exist" Oct 30 22:10:45.029619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.029587 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 22:10:45.031681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.031644 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:10:45.048043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.048002 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2jlsp"] Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155457 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155493 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155648 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155725 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155919 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.155957 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.160391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.156047 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\") pod \"2445cf25-888e-4836-971e-ac2ecd90eea5\" (UID: \"2445cf25-888e-4836-971e-ac2ecd90eea5\") " Oct 30 22:10:45.163098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.163062 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.164043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.163780 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:45.164549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.164236 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.164966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.164697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.167747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.167706 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:45.168739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.168697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:45.169067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.169021 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "kube-api-access-89pgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:45.170683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.170663 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.173762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.173713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data" pod "2445cf25-888e-4836-971e-ac2ecd90eea5" (UID: "2445cf25-888e-4836-971e-ac2ecd90eea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261862 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261894 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261912 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261926 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261941 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261955 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pgh\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-kube-api-access-89pgh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261969 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2445cf25-888e-4836-971e-ac2ecd90eea5-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261982 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2445cf25-888e-4836-971e-ac2ecd90eea5-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.263144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.261997 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2445cf25-888e-4836-971e-ac2ecd90eea5-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.266522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.266195 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:10:45.296930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.296537 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:45.296930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.296593 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:45.296930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.296859 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:45.353432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.353404 5036 scope.go:122] "RemoveContainer" containerID="ad54f10af3fa220ed666a32f5de071f75fc43f6bfd6b2f5252ad1764dee2e345" Oct 30 22:10:45.374511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.374485 5036 scope.go:122] "RemoveContainer" containerID="a7faf7b05b53218dcfc391aa63e42447d2190edaa71cebe539b2e811b305c92d" Oct 30 22:10:45.392828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.392798 5036 scope.go:122] "RemoveContainer" containerID="0b8a4214b88975b91eee158cbf3907e79840b48391641b139517ccbcd83bbd3d" Oct 30 22:10:45.454498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.452066 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2"] Oct 30 22:10:45.464122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.464090 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.464259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.464172 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.464370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.464349 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.464890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.464612 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.464995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.464895 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.465340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.465322 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.466049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.465944 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:45.466319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.466246 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.467146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.467031 5036 scope.go:122] "RemoveContainer" containerID="ab21d505bf066934d35727aa39030afce23b3421c04995b5550038549c7778d9" Oct 30 22:10:45.467146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.466731 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.467605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.467586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.468542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.468489 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:45.468654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.468621 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.468949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.468930 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.469332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.469313 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.469488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.469471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\") pod \"59f2deb7-9007-4a20-9931-319aad1e0c2d\" (UID: \"59f2deb7-9007-4a20-9931-319aad1e0c2d\") " Oct 30 22:10:45.471297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471246 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "kube-api-access-tgn9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:45.471403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471298 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.471403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471329 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.471403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471345 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.471403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471389 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.471403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471404 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.471815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.471418 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2deb7-9007-4a20-9931-319aad1e0c2d-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.472292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.472236 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:45.474541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.473502 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token" pod "59f2deb7-9007-4a20-9931-319aad1e0c2d" (UID: "59f2deb7-9007-4a20-9931-319aad1e0c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:45.488166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.488141 5036 scope.go:122] "RemoveContainer" containerID="ee2356ab6186a8339d5a7e97975599d660c5d4faace7136ef00af29346c2fadc" Oct 30 22:10:45.501245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.501220 5036 scope.go:122] "RemoveContainer" containerID="69b8e1a9991c75e1af281cd0c4f4f664e33618694a9a5ed2c1ac497bc951874d" Oct 30 22:10:45.604348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.581739 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgn9h\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-kube-api-access-tgn9h\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.604348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.581797 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.604348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.581813 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2deb7-9007-4a20-9931-319aad1e0c2d-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:45.636934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.636792 5036 scope.go:122] "RemoveContainer" containerID="f92d05cda3850cac331fe7774146b9b80882b4eec9e5600024c833395b97d8fd" Oct 30 22:10:45.684926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.684900 5036 scope.go:122] "RemoveContainer" containerID="302153d49b702e0908b122eb9f88065a0079d20bafa88da3585e23a7ca5320c8" Oct 30 22:10:45.695817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.695789 5036 scope.go:122] "RemoveContainer" containerID="a4374d19ebf9701ae70c4a9b37656ec6476d6faefacdd53831b79c1d4e0d9097" Oct 30 22:10:45.706947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.706924 5036 scope.go:122] "RemoveContainer" containerID="33bb390a5dcea96ca16bc13a527fd61c7c2791c63fcd2a3173e6d3f882ea97f2" Oct 30 22:10:45.721533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.721509 5036 scope.go:122] "RemoveContainer" containerID="e110ed72b1f832b78f1d65de58c852e691a475bbb92441cdec9b0592885b7b96" Oct 30 22:10:45.954875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.954819 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" event={"ID":"2445cf25-888e-4836-971e-ac2ecd90eea5","Type":"ContainerDied","Data":"b76570a38ec4faf95b8f19be269706e3ab29e0a8ea456970862a40b9cf6c9f11"} Oct 30 22:10:45.955369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.955003 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" event={"ID":"59f2deb7-9007-4a20-9931-319aad1e0c2d","Type":"ContainerDied","Data":"697e7334b08c1e7d1d02b2ee33197100f62ddc72c804b4e10db061ead37d0217"} Oct 30 22:10:45.956751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.956725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f" Oct 30 22:10:45.956897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.956771 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" event={"ID":"a3647372-3e68-429b-ad31-8bad33e78795","Type":"ContainerStarted","Data":"bf239bc8a496467fd2fddccc8695524c6e03ff2510ee45f1c27e276cecae69bc"} Oct 30 22:10:45.957126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:45.957104 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4" Oct 30 22:10:46.046156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.046106 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 22:10:46.055702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.055672 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-kstc4"] Oct 30 22:10:46.067443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.067408 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:10:46.074661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.074636 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-j5s5f"] Oct 30 22:10:46.195368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.195331 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" path="/var/lib/kubelet/pods/2445cf25-888e-4836-971e-ac2ecd90eea5/volumes" Oct 30 22:10:46.195976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.195950 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" path="/var/lib/kubelet/pods/59f2deb7-9007-4a20-9931-319aad1e0c2d/volumes" Oct 30 22:10:46.196491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.196473 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" path="/var/lib/kubelet/pods/5b177cd1-0be4-49f1-b550-6d896570a85b/volumes" Oct 30 22:10:46.295751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.295710 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:46.302692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.301837 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:46.387346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:46.387316 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:47.010041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:47.009776 5036 generic.go:356] "Generic (PLEG): container finished" podID="a3647372-3e68-429b-ad31-8bad33e78795" containerID="65e5319647726ec1c6bf43828c3bb607f34a35f8966026a10b6fd030d49bd702" exitCode=0 Oct 30 22:10:47.010616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:47.010306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" event={"ID":"a3647372-3e68-429b-ad31-8bad33e78795","Type":"ContainerDied","Data":"65e5319647726ec1c6bf43828c3bb607f34a35f8966026a10b6fd030d49bd702"} Oct 30 22:10:47.295897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:47.295787 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:47.295897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:47.295852 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:48.104870 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:10:48.104837 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:10:48.295258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.295222 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:48.295419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.295315 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:48.299789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.299729 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:48.468233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.468194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\") pod \"a3647372-3e68-429b-ad31-8bad33e78795\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " Oct 30 22:10:48.468462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.468366 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\") pod \"a3647372-3e68-429b-ad31-8bad33e78795\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " Oct 30 22:10:48.468462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.468444 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\" (UniqueName: \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\") pod \"a3647372-3e68-429b-ad31-8bad33e78795\" (UID: \"a3647372-3e68-429b-ad31-8bad33e78795\") " Oct 30 22:10:48.468666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.468640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca" pod "a3647372-3e68-429b-ad31-8bad33e78795" (UID: "a3647372-3e68-429b-ad31-8bad33e78795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:48.469089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.469066 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:48.470831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.470804 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca" pod "a3647372-3e68-429b-ad31-8bad33e78795" (UID: "a3647372-3e68-429b-ad31-8bad33e78795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:48.471068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.471039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz" pod "a3647372-3e68-429b-ad31-8bad33e78795" (UID: "a3647372-3e68-429b-ad31-8bad33e78795"). InnerVolumeSpecName "kube-api-access-w6klz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:48.571073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.570990 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3647372-3e68-429b-ad31-8bad33e78795-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:48.571073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:48.571022 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6klz\" (UniqueName: \"kubernetes.io/projected/a3647372-3e68-429b-ad31-8bad33e78795-kube-api-access-w6klz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:49.058452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.058418 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" event={"ID":"a3647372-3e68-429b-ad31-8bad33e78795","Type":"ContainerDied","Data":"bf239bc8a496467fd2fddccc8695524c6e03ff2510ee45f1c27e276cecae69bc"} Oct 30 22:10:49.058673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.058476 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf239bc8a496467fd2fddccc8695524c6e03ff2510ee45f1c27e276cecae69bc" Oct 30 22:10:49.058673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.058539 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2" Oct 30 22:10:49.337340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.332460 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:49.337340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.332523 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:49.667904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.667879 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:10:49.668454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:49.668432 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:10:50.295584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:50.295552 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:10:50.295811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:50.295620 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 22:10:51.300249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:51.300217 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:51.301561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:51.301541 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:10:53.810764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:53.810736 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:10:55.026599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:55.026564 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p"] Oct 30 22:10:55.031883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:55.031848 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5491fcbe-thc4p"] Oct 30 22:10:56.191873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.191838 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699a9764-b4df-4d37-90eb-521d7357a53b" path="/var/lib/kubelet/pods/699a9764-b4df-4d37-90eb-521d7357a53b/volumes" Oct 30 22:10:56.563143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.563036 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:10:56.564239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564221 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" Oct 30 22:10:56.564308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564241 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" Oct 30 22:10:56.564308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564283 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" Oct 30 22:10:56.564308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564291 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" Oct 30 22:10:56.564421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564332 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a3647372-3e68-429b-ad31-8bad33e78795" containerName="revoke-keys" Oct 30 22:10:56.564421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564340 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3647372-3e68-429b-ad31-8bad33e78795" containerName="revoke-keys" Oct 30 22:10:56.564421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564377 5036 cpu_manager.go:414] "containerMap: removing container" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" Oct 30 22:10:56.564421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564384 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" Oct 30 22:10:56.564588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564576 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f2deb7-9007-4a20-9931-319aad1e0c2d" containerName="istio-proxy" Oct 30 22:10:56.564625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564597 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2445cf25-888e-4836-971e-ac2ecd90eea5" containerName="istio-proxy" Oct 30 22:10:56.564625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564610 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b177cd1-0be4-49f1-b550-6d896570a85b" containerName="istio-proxy" Oct 30 22:10:56.564687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.564624 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3647372-3e68-429b-ad31-8bad33e78795" containerName="revoke-keys" Oct 30 22:10:56.569448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.569428 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.595487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.595458 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a1ea1e-openshift-default-dockercfg-wfdmd\"" Oct 30 22:10:56.608760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.608736 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:10:56.641742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.641712 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.641901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.641879 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.641999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642105 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjnz\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642147 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642281 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.642563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.642539 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.763676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.751337 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.790290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.764282 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.790642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.790608 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.793936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.793908 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjnz\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.801567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.801537 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.802319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.802293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.802912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.802892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.809000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.808973 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.821749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.821690 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.831356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.754738 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.831956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.821488 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.832147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.802130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.832279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.822527 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:56.832400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:56.803320 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.047999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.047930 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.049093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.048877 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.066686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.065947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.066686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.066543 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjnz\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\") pod \"e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.187291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.186544 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:57.686483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.686452 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:10:57.686898 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:10:57.686820 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec71c40_29db_4de4_b757_3c5d5a6573e4.slice/crio-b74187c614a800d5e8942f1268ed60e47989c1c16b303b675455325128a945a4 WatchSource:0}: Error finding container b74187c614a800d5e8942f1268ed60e47989c1c16b303b675455325128a945a4: Status 404 returned error can't find the container with id b74187c614a800d5e8942f1268ed60e47989c1c16b303b675455325128a945a4 Oct 30 22:10:57.689855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.689799 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:57.689933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.689887 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:57.689933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:57.689918 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:58.177020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.176986 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerStarted","Data":"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74"} Oct 30 22:10:58.177225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.177041 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerStarted","Data":"b74187c614a800d5e8942f1268ed60e47989c1c16b303b675455325128a945a4"} Oct 30 22:10:58.196311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.196258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:58.200972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.200938 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:58.205179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.205121 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podStartSLOduration=2.20510414 podStartE2EDuration="2.20510414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:58.198902693 +0000 UTC m=+1882.148857650" watchObservedRunningTime="2026-10-30 22:10:58.20510414 +0000 UTC m=+1882.155059088" Oct 30 22:10:58.698614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:58.698576 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:10:59.185611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:59.185579 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:10:59.186412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:10:59.186389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:11:05.525294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.521700 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:11:05.616385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.613601 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.652291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.647699 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f0f5e2\"" Oct 30 22:11:05.677293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.671703 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f0f5e2-dockercfg-qtsxg\"" Oct 30 22:11:05.760331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.751211 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:11:05.849357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.834462 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgr7\" (UniqueName: \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.849357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.834559 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.946311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.945578 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.946311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.945979 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgr7\" (UniqueName: \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.947241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.947215 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:05.961966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:05.961936 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgr7\" (UniqueName: \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\") pod \"payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:06.159886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:06.159852 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:06.681803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:06.681769 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:11:06.682192 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:11:06.682036 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f6f7dc_5a7a_4fc3_897c_95e016fe46c1.slice/crio-3ce8d67f01f86a95f1b54be9f4c5a5082ed0fddcb89df9dbdb1bc96d7cc4c6b5 WatchSource:0}: Error finding container 3ce8d67f01f86a95f1b54be9f4c5a5082ed0fddcb89df9dbdb1bc96d7cc4c6b5: Status 404 returned error can't find the container with id 3ce8d67f01f86a95f1b54be9f4c5a5082ed0fddcb89df9dbdb1bc96d7cc4c6b5 Oct 30 22:11:07.664608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:07.663471 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" event={"ID":"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1","Type":"ContainerStarted","Data":"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1"} Oct 30 22:11:07.664608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:07.663528 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" event={"ID":"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1","Type":"ContainerStarted","Data":"3ce8d67f01f86a95f1b54be9f4c5a5082ed0fddcb89df9dbdb1bc96d7cc4c6b5"} Oct 30 22:11:07.664608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:07.664392 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:07.686538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:07.686484 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" podStartSLOduration=2.686471187 podStartE2EDuration="2.686471187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:07.683837348 +0000 UTC m=+1891.633792309" watchObservedRunningTime="2026-10-30 22:11:07.686471187 +0000 UTC m=+1891.636426132" Oct 30 22:11:13.739661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:13.739615 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" podUID="ade4040b-5424-47dc-b285-5f84ae014ef1" containerName="main" containerID="cri-o://55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57" gracePeriod=45 Oct 30 22:11:14.166529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.166501 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:11:14.282324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282249 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.282527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282411 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.282527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282457 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:14.282653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282570 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.282723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282696 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\" (UniqueName: \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.282839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282817 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:14.282912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282859 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.282912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.282895 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\") pod \"ade4040b-5424-47dc-b285-5f84ae014ef1\" (UID: \"ade4040b-5424-47dc-b285-5f84ae014ef1\") " Oct 30 22:11:14.283212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.283065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:14.283807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.283787 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.283893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.283813 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.283893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.283828 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.285050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.285029 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:14.285132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.285109 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:14.285188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.285148 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc" pod "ade4040b-5424-47dc-b285-5f84ae014ef1" (UID: "ade4040b-5424-47dc-b285-5f84ae014ef1"). InnerVolumeSpecName "kube-api-access-lz2lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:14.385650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.385624 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade4040b-5424-47dc-b285-5f84ae014ef1-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.385650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.385650 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz2lc\" (UniqueName: \"kubernetes.io/projected/ade4040b-5424-47dc-b285-5f84ae014ef1-kube-api-access-lz2lc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.385832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.385663 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade4040b-5424-47dc-b285-5f84ae014ef1-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:14.838870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.838774 5036 generic.go:356] "Generic (PLEG): container finished" podID="ade4040b-5424-47dc-b285-5f84ae014ef1" containerID="55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57" exitCode=0 Oct 30 22:11:14.838870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.838833 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" Oct 30 22:11:14.839490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.838863 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" event={"ID":"ade4040b-5424-47dc-b285-5f84ae014ef1","Type":"ContainerDied","Data":"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57"} Oct 30 22:11:14.839490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.838922 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh" event={"ID":"ade4040b-5424-47dc-b285-5f84ae014ef1","Type":"ContainerDied","Data":"e3917022cdce6c6cfdab3257ed1666dd34d53b6db79e99ec70fe97c2199f3e7b"} Oct 30 22:11:14.839490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.838945 5036 scope.go:122] "RemoveContainer" containerID="55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57" Oct 30 22:11:14.853592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.853572 5036 scope.go:122] "RemoveContainer" containerID="55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57" Oct 30 22:11:14.853929 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:11:14.853911 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57\": container with ID starting with 55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57 not found: ID does not exist" containerID="55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57" Oct 30 22:11:14.854003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.853943 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57"} err="failed to get container status \"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57\": rpc error: code = NotFound desc = could not find container \"55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57\": container with ID starting with 55f5b467108cff9b8805dce1e5a6ecc791bcb911af11ad1f588c11842a354e57 not found: ID does not exist" Oct 30 22:11:14.868781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.868757 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:11:14.874929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:14.874910 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c8ecf/auth-test-model-9c8ecf-kserve-66c54dcdc7-8jkvh"] Oct 30 22:11:16.192770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:16.192737 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade4040b-5424-47dc-b285-5f84ae014ef1" path="/var/lib/kubelet/pods/ade4040b-5424-47dc-b285-5f84ae014ef1/volumes" Oct 30 22:11:18.210187 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:11:18.210143 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:11:18.661339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.651492 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:11:18.665288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.661791 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ade4040b-5424-47dc-b285-5f84ae014ef1" containerName="main" Oct 30 22:11:18.665288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.661824 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade4040b-5424-47dc-b285-5f84ae014ef1" containerName="main" Oct 30 22:11:18.665288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.662053 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade4040b-5424-47dc-b285-5f84ae014ef1" containerName="main" Oct 30 22:11:18.680291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.679958 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.693308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.692754 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-813840a060cc\"" Oct 30 22:11:18.696296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.693571 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\"" Oct 30 22:11:18.696296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.693628 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\"" Oct 30 22:11:18.696296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.693706 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-zdlx9\"" Oct 30 22:11:18.696296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.693754 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\"" Oct 30 22:11:18.696296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.693807 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\"" Oct 30 22:11:18.721400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.715477 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:11:18.721400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.717719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.721400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.717835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4rk\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.721400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.717923 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.721400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.718084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.820795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.820754 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.822152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.822125 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4rk\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.822418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.822399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.822679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.822661 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.823534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.823510 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.823671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.821967 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.853789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.853723 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:18.923431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:18.914054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4rk\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:19.036710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:19.036664 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:19.465814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:19.465784 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:11:19.468159 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:11:19.467607 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df23b6f_fcde_40e2_8cd3_db75e8a0d55b.slice/crio-a259a92f639d92563bff21448f3f23dd3e0a6b9c141d16bf994e80d5684ce390 WatchSource:0}: Error finding container a259a92f639d92563bff21448f3f23dd3e0a6b9c141d16bf994e80d5684ce390: Status 404 returned error can't find the container with id a259a92f639d92563bff21448f3f23dd3e0a6b9c141d16bf994e80d5684ce390 Oct 30 22:11:19.723843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:19.723745 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:11:20.002586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:20.002487 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" event={"ID":"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b","Type":"ContainerStarted","Data":"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37"} Oct 30 22:11:20.002586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:20.002557 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" event={"ID":"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b","Type":"ContainerStarted","Data":"a259a92f639d92563bff21448f3f23dd3e0a6b9c141d16bf994e80d5684ce390"} Oct 30 22:11:20.002774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:20.002588 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:20.030354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:20.030301 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" podStartSLOduration=2.030287306 podStartE2EDuration="2.030287306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:20.025659198 +0000 UTC m=+1903.975614145" watchObservedRunningTime="2026-10-30 22:11:20.030287306 +0000 UTC m=+1903.980242249" Oct 30 22:11:31.016284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:31.016235 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:11:44.903288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:44.902351 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:11:45.236794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.236708 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:11:45.238055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.238032 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.260909 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"openshift-service-ca.crt\"" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.261325 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a1ea1e\"" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.261634 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"payload-processing-external-model-e2e--6ed6d0c0-dockercfg-r9rd4\"" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.261793 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"agc-cred-c5c8ee21f27c\"" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.262053 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"kube-root-ca.crt\"" Oct 30 22:11:45.266291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.262378 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"routing-overlay\"" Oct 30 22:11:45.362699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.362519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.362699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.362644 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.362948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.362835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.363107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.363084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6p97\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.464446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.464402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.464624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.464607 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p97\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.464730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.464713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.464799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.464789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.465375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.465346 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.465567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.465454 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.467499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.467479 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.478732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.478701 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p97\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.592807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.592723 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:45.908081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.908048 5036 scope.go:122] "RemoveContainer" containerID="eb5460b26a3e74ecbd4fd2ab9f41d75c779518920b080604581d62fadcfc5c94" Oct 30 22:11:45.979064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:45.979040 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:11:45.982602 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:11:45.982569 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5233cbcb_e390_4a37_90dc_e022bd586c56.slice/crio-cc67fd19903cd5a7ce5f37d01facf6dfae7d0d9766eb1ffe1fdabc28e25976bf WatchSource:0}: Error finding container cc67fd19903cd5a7ce5f37d01facf6dfae7d0d9766eb1ffe1fdabc28e25976bf: Status 404 returned error can't find the container with id cc67fd19903cd5a7ce5f37d01facf6dfae7d0d9766eb1ffe1fdabc28e25976bf Oct 30 22:11:46.534788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:46.534691 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" event={"ID":"5233cbcb-e390-4a37-90dc-e022bd586c56","Type":"ContainerStarted","Data":"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9"} Oct 30 22:11:46.534788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:46.534762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" event={"ID":"5233cbcb-e390-4a37-90dc-e022bd586c56","Type":"ContainerStarted","Data":"cc67fd19903cd5a7ce5f37d01facf6dfae7d0d9766eb1ffe1fdabc28e25976bf"} Oct 30 22:11:46.534788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:46.534792 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:46.567498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:46.567444 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" podStartSLOduration=2.567426964 podStartE2EDuration="2.567426964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:46.552645154 +0000 UTC m=+1930.502600103" watchObservedRunningTime="2026-10-30 22:11:46.567426964 +0000 UTC m=+1930.517381909" Oct 30 22:11:48.125636 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:11:48.125546 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:11:51.598907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.598876 5036 generic.go:356] "Generic (PLEG): container finished" podID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerID="cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74" exitCode=0 Oct 30 22:11:51.599469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.598952 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerDied","Data":"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74"} Oct 30 22:11:51.599605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.599586 5036 scope.go:122] "RemoveContainer" containerID="cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74" Oct 30 22:11:51.601147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.601123 5036 generic.go:356] "Generic (PLEG): container finished" podID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerID="4d14ca2a54c8528d009e7cd8975093a70a298de7052ffbc858b8a70a9a63c368" exitCode=0 Oct 30 22:11:51.601311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.601153 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerDied","Data":"4d14ca2a54c8528d009e7cd8975093a70a298de7052ffbc858b8a70a9a63c368"} Oct 30 22:11:51.601588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.601573 5036 scope.go:122] "RemoveContainer" containerID="4d14ca2a54c8528d009e7cd8975093a70a298de7052ffbc858b8a70a9a63c368" Oct 30 22:11:51.601930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.601899 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.602000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.601984 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.602046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.602033 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.603386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.603360 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.603469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.603425 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.603469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:51.603457 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:52.621572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:52.621535 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerStarted","Data":"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be"} Oct 30 22:11:52.624359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:52.624326 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerStarted","Data":"baeac5fba68dc44b403da9beec9dc59042f194791237fcfaf0e2f0c25856b7bc"} Oct 30 22:11:52.664466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:52.663244 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:11:52.665303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:52.665239 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:52.665458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:52.665340 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:53.190295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.188633 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:53.190295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.188694 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:53.190295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.189116 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:11:53.604366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.604250 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:11:53.661293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.660878 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:53.661293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:53.660938 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:54.188765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:54.188731 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:54.188992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:54.188790 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:54.683286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:54.682654 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:54.694496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:54.694454 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:55.188755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:55.188716 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:55.188970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:55.188776 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:55.677590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:55.677549 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:55.677784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:55.677614 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:56.202294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:56.198431 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:56.202294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:56.198500 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:56.667291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:56.666748 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:56.667291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:56.666822 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:57.188753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:57.188715 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:57.188945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:57.188772 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:57.581344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:57.581190 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:11:57.681292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:57.680589 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:57.681292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:57.680660 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:58.188324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:58.188289 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 30 22:11:58.188552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:58.188347 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 30 22:11:58.665722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:58.665675 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 22:11:58.666137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:58.665760 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 22:11:59.186369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:59.186338 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:11:59.194350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:59.194328 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:11:59.689468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:59.689433 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:11:59.710293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:11:59.704752 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:12:00.061353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:00.061241 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:12:01.778815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.778773 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:12:01.785471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.785447 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:01.792762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.792678 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-e2e--fed5953e-dockercfg-tbl9w\"" Oct 30 22:12:01.792911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.792788 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-26ab1b64b723\"" Oct 30 22:12:01.793816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.793163 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"openshift-service-ca.crt\"" Oct 30 22:12:01.793816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.793212 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a5f192\"" Oct 30 22:12:01.793816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.793213 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"kube-root-ca.crt\"" Oct 30 22:12:01.793816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.793330 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"routing-overlay\"" Oct 30 22:12:01.800982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.800956 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:12:01.937310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.933726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:01.937310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.933860 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:01.937310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.933905 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:01.937310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:01.933986 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x2z\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.021311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.020339 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j"] Oct 30 22:12:02.057616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.057536 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.082598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.082564 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:02.099559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.086562 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:12:02.099804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.086581 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:02.099927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.086621 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:02.100033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.095324 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j"] Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230285 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58px\" (UniqueName: \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230611 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x2z\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.232309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.230681 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.255700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.246344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.255700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.247095 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.255700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.254188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.272222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.270651 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x2z\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.332464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.332367 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.332638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.332556 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.332638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.332581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58px\" (UniqueName: \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.333358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.333329 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.335082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.335056 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.345370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.345343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58px\" (UniqueName: \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\") pod \"maas-api-revoke-keys-e2e-autores-f0f5e2-6547j\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.410532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.410503 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:02.557522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.557481 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:02.702863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.702829 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:12:02.704130 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:12:02.704104 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b1cb46_c848_422e_bc00_54f0acb6a290.slice/crio-87934ae611fdc1d9c2fdc5d807720ca0c43bbaaa8e74d77b7ab4070532534115 WatchSource:0}: Error finding container 87934ae611fdc1d9c2fdc5d807720ca0c43bbaaa8e74d77b7ab4070532534115: Status 404 returned error can't find the container with id 87934ae611fdc1d9c2fdc5d807720ca0c43bbaaa8e74d77b7ab4070532534115 Oct 30 22:12:02.861595 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:12:02.861566 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2528534e_2cb1_4b2a_abe7_6103964b2634.slice/crio-dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4 WatchSource:0}: Error finding container dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4: Status 404 returned error can't find the container with id dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4 Oct 30 22:12:02.862821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:02.862785 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j"] Oct 30 22:12:03.270862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.270822 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" event={"ID":"f5b1cb46-c848-422e-bc00-54f0acb6a290","Type":"ContainerStarted","Data":"bc5d527fbbcdb0139175318e924ffa0812049156b560fccb5c1d4a7f6bc0c48b"} Oct 30 22:12:03.271044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.270882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" event={"ID":"f5b1cb46-c848-422e-bc00-54f0acb6a290","Type":"ContainerStarted","Data":"87934ae611fdc1d9c2fdc5d807720ca0c43bbaaa8e74d77b7ab4070532534115"} Oct 30 22:12:03.271044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.270907 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:03.273011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.272921 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" event={"ID":"2528534e-2cb1-4b2a-abe7-6103964b2634","Type":"ContainerStarted","Data":"33073f856f609d9e4a7a78f2e4ca6d1c05c3da3c19357371b689870f2f3ad7cd"} Oct 30 22:12:03.273011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.272978 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" event={"ID":"2528534e-2cb1-4b2a-abe7-6103964b2634","Type":"ContainerStarted","Data":"dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4"} Oct 30 22:12:03.293241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.293194 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" podStartSLOduration=2.293181794 podStartE2EDuration="2.293181794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:03.28867833 +0000 UTC m=+1947.238633276" watchObservedRunningTime="2026-10-30 22:12:03.293181794 +0000 UTC m=+1947.243136739" Oct 30 22:12:03.308961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:03.308917 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" podStartSLOduration=2.308903149 podStartE2EDuration="2.308903149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:03.30472056 +0000 UTC m=+1947.254675508" watchObservedRunningTime="2026-10-30 22:12:03.308903149 +0000 UTC m=+1947.258858095" Oct 30 22:12:04.287311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:04.287223 5036 generic.go:356] "Generic (PLEG): container finished" podID="2528534e-2cb1-4b2a-abe7-6103964b2634" containerID="33073f856f609d9e4a7a78f2e4ca6d1c05c3da3c19357371b689870f2f3ad7cd" exitCode=0 Oct 30 22:12:04.287854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:04.287338 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" event={"ID":"2528534e-2cb1-4b2a-abe7-6103964b2634","Type":"ContainerDied","Data":"33073f856f609d9e4a7a78f2e4ca6d1c05c3da3c19357371b689870f2f3ad7cd"} Oct 30 22:12:05.496845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.496817 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:05.604197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.604096 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\" (UniqueName: \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\") pod \"2528534e-2cb1-4b2a-abe7-6103964b2634\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " Oct 30 22:12:05.604377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.604259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\") pod \"2528534e-2cb1-4b2a-abe7-6103964b2634\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " Oct 30 22:12:05.604489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.604471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\") pod \"2528534e-2cb1-4b2a-abe7-6103964b2634\" (UID: \"2528534e-2cb1-4b2a-abe7-6103964b2634\") " Oct 30 22:12:05.604900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.604866 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca" pod "2528534e-2cb1-4b2a-abe7-6103964b2634" (UID: "2528534e-2cb1-4b2a-abe7-6103964b2634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:05.605639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.605614 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:05.606756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.606720 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca" pod "2528534e-2cb1-4b2a-abe7-6103964b2634" (UID: "2528534e-2cb1-4b2a-abe7-6103964b2634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:05.606998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.606981 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px" pod "2528534e-2cb1-4b2a-abe7-6103964b2634" (UID: "2528534e-2cb1-4b2a-abe7-6103964b2634"). InnerVolumeSpecName "kube-api-access-t58px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:05.707302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.707249 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58px\" (UniqueName: \"kubernetes.io/projected/2528534e-2cb1-4b2a-abe7-6103964b2634-kube-api-access-t58px\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:05.707302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:05.707303 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2528534e-2cb1-4b2a-abe7-6103964b2634-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:06.310994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:06.310963 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" Oct 30 22:12:06.311180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:06.310991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j" event={"ID":"2528534e-2cb1-4b2a-abe7-6103964b2634","Type":"ContainerDied","Data":"dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4"} Oct 30 22:12:06.311180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:06.311045 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc6afc859109c73e98e95ec2b64cb5a4efed3e29a96d4ece637799f24d2d9be4" Oct 30 22:12:12.120059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:12.120020 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:12:14.290829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:14.290799 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:12:15.880964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:15.880926 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j"] Oct 30 22:12:15.886547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:15.886522 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0f5e2-6547j"] Oct 30 22:12:16.191018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:16.190981 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2528534e-2cb1-4b2a-abe7-6103964b2634" path="/var/lib/kubelet/pods/2528534e-2cb1-4b2a-abe7-6103964b2634/volumes" Oct 30 22:12:17.120463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.120415 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" podUID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" containerName="payload-processing" containerID="cri-o://76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1" gracePeriod=25 Oct 30 22:12:17.450520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.450485 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:12:17.477225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.477189 5036 generic.go:356] "Generic (PLEG): container finished" podID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" containerID="76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1" exitCode=0 Oct 30 22:12:17.477537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.477316 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" Oct 30 22:12:17.477697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.477316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" event={"ID":"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1","Type":"ContainerDied","Data":"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1"} Oct 30 22:12:17.478632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.478606 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g" event={"ID":"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1","Type":"ContainerDied","Data":"3ce8d67f01f86a95f1b54be9f4c5a5082ed0fddcb89df9dbdb1bc96d7cc4c6b5"} Oct 30 22:12:17.478739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.478655 5036 scope.go:122] "RemoveContainer" containerID="76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1" Oct 30 22:12:17.492576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.492538 5036 scope.go:122] "RemoveContainer" containerID="76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1" Oct 30 22:12:17.492978 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:12:17.492955 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1\": container with ID starting with 76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1 not found: ID does not exist" containerID="76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1" Oct 30 22:12:17.493080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.492987 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1"} err="failed to get container status \"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1\": rpc error: code = NotFound desc = could not find container \"76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1\": container with ID starting with 76e2dd72770066164592b177852a2eec44cfc4ec16242c2516f3af557ba792d1 not found: ID does not exist" Oct 30 22:12:17.512527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.512233 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\" (UniqueName: \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\") pod \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " Oct 30 22:12:17.513689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.513150 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\") pod \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\" (UID: \"35f6f7dc-5a7a-4fc3-897c-95e016fe46c1\") " Oct 30 22:12:17.513820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.513701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume" pod "35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" (UID: "35f6f7dc-5a7a-4fc3-897c-95e016fe46c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:17.514527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.514493 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:17.515344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.515319 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7" pod "35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" (UID: "35f6f7dc-5a7a-4fc3-897c-95e016fe46c1"). InnerVolumeSpecName "kube-api-access-8jgr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:17.628301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.628197 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgr7\" (UniqueName: \"kubernetes.io/projected/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1-kube-api-access-8jgr7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:17.803821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.803792 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:12:17.833045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:17.833007 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0f5e2-6d589845d9-28x9g"] Oct 30 22:12:18.192968 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:12:18.192925 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:12:18.206251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:18.206215 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" path="/var/lib/kubelet/pods/35f6f7dc-5a7a-4fc3-897c-95e016fe46c1/volumes" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.184071 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185231 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2528534e-2cb1-4b2a-abe7-6103964b2634" containerName="revoke-keys" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185253 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2528534e-2cb1-4b2a-abe7-6103964b2634" containerName="revoke-keys" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185314 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" containerName="payload-processing" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185323 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" containerName="payload-processing" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185554 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2528534e-2cb1-4b2a-abe7-6103964b2634" containerName="revoke-keys" Oct 30 22:12:25.191324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.185570 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f6f7dc-5a7a-4fc3-897c-95e016fe46c1" containerName="payload-processing" Oct 30 22:12:25.192780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.192353 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.206544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.199185 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6f8222-openshift-default-dockercfg-ntt7c\"" Oct 30 22:12:25.209320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.209249 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:12:25.322288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322177 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322295 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322329 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322389 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322503 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322624 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322724 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.322904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.322876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97kb\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488230 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488353 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488477 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488591 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488716 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.490205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.488970 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97kb\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.512337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.497493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.512337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.497766 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.512337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.497995 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.512337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.498634 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.538287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.530756 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.580290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.570904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.606286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.595612 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.606286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.597490 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97kb\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.656937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.656840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\") pod \"e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:25.829538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:25.829454 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:26.143788 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:12:26.143761 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdcb908_5eb3_4825_91c5_0e3884697059.slice/crio-5d65f3b805fefd456e1b1db1c32a53d37ebe9707e2775b75401be043884e9b21 WatchSource:0}: Error finding container 5d65f3b805fefd456e1b1db1c32a53d37ebe9707e2775b75401be043884e9b21: Status 404 returned error can't find the container with id 5d65f3b805fefd456e1b1db1c32a53d37ebe9707e2775b75401be043884e9b21 Oct 30 22:12:26.144158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.144133 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:12:26.150808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.147530 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:26.150808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.147700 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:26.150808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.147805 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:26.662661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.662617 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerStarted","Data":"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4"} Oct 30 22:12:26.663072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.662678 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerStarted","Data":"5d65f3b805fefd456e1b1db1c32a53d37ebe9707e2775b75401be043884e9b21"} Oct 30 22:12:26.691055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.691008 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podStartSLOduration=1.690995603 podStartE2EDuration="1.690995603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:26.688245929 +0000 UTC m=+1970.638200898" watchObservedRunningTime="2026-10-30 22:12:26.690995603 +0000 UTC m=+1970.640950549" Oct 30 22:12:26.837288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:26.829729 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:27.837438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:27.837347 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:28.703931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:28.703900 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:28.705192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:28.705172 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:12:30.073354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:30.073081 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6"] Oct 30 22:12:30.082743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:30.082131 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81503875-w4vz6"] Oct 30 22:12:30.207869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:30.207827 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf52da4-2011-4749-8652-a2a49dd10613" path="/var/lib/kubelet/pods/ccf52da4-2011-4749-8652-a2a49dd10613/volumes" Oct 30 22:12:40.068396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:40.068328 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5"] Oct 30 22:12:40.093871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:40.088885 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1310d2dc-vbmf5"] Oct 30 22:12:40.190371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:40.190339 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f164e9d1-b7c1-4cd9-ba08-8357ee967694" path="/var/lib/kubelet/pods/f164e9d1-b7c1-4cd9-ba08-8357ee967694/volumes" Oct 30 22:12:45.968183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:45.968148 5036 scope.go:122] "RemoveContainer" containerID="55d7a6d951de0aee9790cc42573565e4a3f69d26e049bba70a1bd39162d17cd7" Oct 30 22:12:48.103212 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:12:48.103173 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:12:52.263461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.263426 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:12:52.270679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.270652 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.318557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.318525 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:12:52.329832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.329802 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-3261044a2004\"" Oct 30 22:12:52.330005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.329844 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-plugins-e2e-extmod-b-26a128\"" Oct 30 22:12:52.330516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.330492 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-953110ac8044\"" Oct 30 22:12:52.330713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.330505 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-e2e--a9381736-dockercfg-kqh8l\"" Oct 30 22:12:52.330713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.330523 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"kube-root-ca.crt\"" Oct 30 22:12:52.330929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.330594 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"routing-overlay\"" Oct 30 22:12:52.330929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.330654 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"openshift-service-ca.crt\"" Oct 30 22:12:52.363220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.363187 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.363419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.363398 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.363543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.363452 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6p9\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.363639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.363614 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.465822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.465794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.465988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.465834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6p9\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.465988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.465898 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.466133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.466112 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.466753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.466725 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.467016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.466991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.468914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.468891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.481501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.481470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6p9\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.586158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.586066 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:52.990369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:52.990330 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:12:53.190291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.186836 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" event={"ID":"abf61fc1-d7a7-4c4a-9534-c849b8daf88f","Type":"ContainerStarted","Data":"77f0e3d1eb806b4a3d3bf9dc403adb180e8da966c9e5fda29a5317e48e85d8dc"} Oct 30 22:12:53.233592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.233556 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:12:53.241088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.241057 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.245435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.245367 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-3aae80dfab12\"" Oct 30 22:12:53.248986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.248964 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:12:53.396598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.396564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.397043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.396635 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.397043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.396806 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrgh\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.397043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.397017 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.498187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.498109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrgh\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.498391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.498244 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.498465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.498394 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.498465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.498447 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.498975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.498948 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.499186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.499164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.501140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.501120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.510510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.510483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrgh\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.571760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.571714 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:53.889586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:53.889532 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:12:53.890909 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:12:53.890879 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb1bdc8_564f_4f19_814c_ecfec7287dee.slice/crio-b1c31dfdf28d56241353060902b335e8043efa7ba17f31178bb56c60831f0f1f WatchSource:0}: Error finding container b1c31dfdf28d56241353060902b335e8043efa7ba17f31178bb56c60831f0f1f: Status 404 returned error can't find the container with id b1c31dfdf28d56241353060902b335e8043efa7ba17f31178bb56c60831f0f1f Oct 30 22:12:54.199501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.199391 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" event={"ID":"2bb1bdc8-564f-4f19-814c-ecfec7287dee","Type":"ContainerStarted","Data":"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252"} Oct 30 22:12:54.199501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.199460 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:12:54.199501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.199486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" event={"ID":"2bb1bdc8-564f-4f19-814c-ecfec7287dee","Type":"ContainerStarted","Data":"b1c31dfdf28d56241353060902b335e8043efa7ba17f31178bb56c60831f0f1f"} Oct 30 22:12:54.201300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.201256 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" event={"ID":"abf61fc1-d7a7-4c4a-9534-c849b8daf88f","Type":"ContainerStarted","Data":"d2f5af6a6d8ea39e41f325ed2752bbc042c01935a56baec077e1af5ee06f8867"} Oct 30 22:12:54.201430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.201329 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:12:54.225706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.225659 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" podStartSLOduration=1.225647206 podStartE2EDuration="1.225647206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:54.221815099 +0000 UTC m=+1998.171770046" watchObservedRunningTime="2026-10-30 22:12:54.225647206 +0000 UTC m=+1998.175602158" Oct 30 22:12:54.244049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:12:54.243998 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" podStartSLOduration=2.243984827 podStartE2EDuration="2.243984827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:54.240312367 +0000 UTC m=+1998.190267313" watchObservedRunningTime="2026-10-30 22:12:54.243984827 +0000 UTC m=+1998.193939773" Oct 30 22:13:05.225556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:05.225494 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:13:05.226913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:05.226887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:13:05.335093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:05.335051 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:13:06.205184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.205153 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:13:06.210553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.210520 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.229782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.220317 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d8c835-openshift-default-dockercfg-bf6v4\"" Oct 30 22:13:06.230585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.230381 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390605 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878bz\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390678 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390768 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.390903 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.391005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.391363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.391087 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492648 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492735 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878bz\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492800 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492891 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.492975 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.493022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.493102 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.493182 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.495339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.494797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.518286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.512777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.544585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.544552 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.545532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.545502 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.549046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.548530 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.567116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.567070 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.606706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.578588 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.606706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.593389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-878bz\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.678296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.663124 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\") pod \"e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:06.832984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:06.831725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:07.197225 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:07.197192 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fad8555_d2ee_49ec_a2a7_142f5ab22c10.slice/crio-ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb WatchSource:0}: Error finding container ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb: Status 404 returned error can't find the container with id ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb Oct 30 22:13:07.197781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.197697 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:13:07.200123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.200087 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:07.200252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.200150 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:07.200252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.200182 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:07.363647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.363600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerStarted","Data":"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b"} Oct 30 22:13:07.364114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.363668 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerStarted","Data":"ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb"} Oct 30 22:13:07.386259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.386201 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podStartSLOduration=1.386184159 podStartE2EDuration="1.386184159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:07.385853485 +0000 UTC m=+2011.335808462" watchObservedRunningTime="2026-10-30 22:13:07.386184159 +0000 UTC m=+2011.336139107" Oct 30 22:13:07.839295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:07.838357 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:08.841358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:08.841326 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:09.422300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:09.419929 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:09.429298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:09.426459 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:10.335936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.335876 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" podUID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" containerName="payload-processing" containerID="cri-o://6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37" gracePeriod=25 Oct 30 22:13:10.637327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.637254 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:13:10.718655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.718621 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\") pod \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " Oct 30 22:13:10.718880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.718735 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\") pod \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " Oct 30 22:13:10.718880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.718801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\") pod \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " Oct 30 22:13:10.718880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.718855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\") pod \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\" (UID: \"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b\") " Oct 30 22:13:10.719944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.719918 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume" pod "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" (UID: "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:10.720084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.720033 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.720397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.720369 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume" pod "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" (UID: "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:10.723143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.723109 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk" pod "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" (UID: "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b"). InnerVolumeSpecName "kube-api-access-4g4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:10.725416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.725387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials" pod "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" (UID: "2df23b6f-fcde-40e2-8cd3-db75e8a0d55b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:10.822988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.822951 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4rk\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-kube-api-access-4g4rk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.822988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.822985 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.823188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:10.822998 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:11.440773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.440731 5036 generic.go:356] "Generic (PLEG): container finished" podID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" containerID="6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37" exitCode=0 Oct 30 22:13:11.441260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.440777 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" Oct 30 22:13:11.441260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.440806 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" event={"ID":"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b","Type":"ContainerDied","Data":"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37"} Oct 30 22:13:11.441260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.440862 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz" event={"ID":"2df23b6f-fcde-40e2-8cd3-db75e8a0d55b","Type":"ContainerDied","Data":"a259a92f639d92563bff21448f3f23dd3e0a6b9c141d16bf994e80d5684ce390"} Oct 30 22:13:11.441260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.440885 5036 scope.go:122] "RemoveContainer" containerID="6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37" Oct 30 22:13:11.456437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.456407 5036 scope.go:122] "RemoveContainer" containerID="6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37" Oct 30 22:13:11.456798 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:11.456770 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37\": container with ID starting with 6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37 not found: ID does not exist" containerID="6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37" Oct 30 22:13:11.456872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.456813 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37"} err="failed to get container status \"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37\": rpc error: code = NotFound desc = could not find container \"6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37\": container with ID starting with 6eea8cee745f5e2b4456bd35251a9cfe1385b50309cba9bc161c277048ce1e37 not found: ID does not exist" Oct 30 22:13:11.471611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.471581 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:13:11.483549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:11.483512 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8gcxgz"] Oct 30 22:13:12.238330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.235818 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" path="/var/lib/kubelet/pods/2df23b6f-fcde-40e2-8cd3-db75e8a0d55b/volumes" Oct 30 22:13:12.291085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.291044 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:13:12.292666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.292631 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" containerName="payload-processing" Oct 30 22:13:12.292666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.292666 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" containerName="payload-processing" Oct 30 22:13:12.292927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.292909 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df23b6f-fcde-40e2-8cd3-db75e8a0d55b" containerName="payload-processing" Oct 30 22:13:12.304590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.304556 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:13:12.304849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.304775 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.308709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.308223 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"agc-cred-c7a7a72d9d8e\"" Oct 30 22:13:12.421923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.421888 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.422164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.421968 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.422586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.422553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtlc\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.423001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.422968 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.524937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.524724 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.524937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.524812 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.525805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.525696 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.525955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.525894 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtlc\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.526626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.526602 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.527406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.527358 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.529054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.529031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.541485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.541461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtlc\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:12.630968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:12.630931 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:13.156546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.156513 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:13:13.159086 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:13.159052 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fd4a0_9c5e_4367_828c_cef3ac3c48e1.slice/crio-39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc WatchSource:0}: Error finding container 39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc: Status 404 returned error can't find the container with id 39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc Oct 30 22:13:13.315385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.315351 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:13.320599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.320576 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.327817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.327524 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"openshift-service-ca.crt\"" Oct 30 22:13:13.327817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.327781 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9d868\"" Oct 30 22:13:13.328088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.327844 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"routing-overlay\"" Oct 30 22:13:13.328088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.327942 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"kube-root-ca.crt\"" Oct 30 22:13:13.332794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.332771 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-ac35153a2219\"" Oct 30 22:13:13.336648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.336622 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:13.414506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.414414 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfw6\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.414506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.414494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.414741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.414716 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.414920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.414901 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.464074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.464039 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" event={"ID":"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1","Type":"ContainerStarted","Data":"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e"} Oct 30 22:13:13.464227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.464092 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" event={"ID":"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1","Type":"ContainerStarted","Data":"39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc"} Oct 30 22:13:13.464227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.464156 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:13.486778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.486722 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" podStartSLOduration=1.486708646 podStartE2EDuration="1.486708646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:13.483430628 +0000 UTC m=+2017.433385575" watchObservedRunningTime="2026-10-30 22:13:13.486708646 +0000 UTC m=+2017.436663591" Oct 30 22:13:13.516716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.516685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.516943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.516798 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfw6\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.516943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.516844 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.517111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.517080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.517669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.517644 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.517932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.517908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.520036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.520013 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.529403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.529379 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfw6\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.641969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.641932 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:13.642756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.642729 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:13:13.820509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.820475 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:13:13.903292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.827976 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:13.903292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.894479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd96\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:13.903292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.903102 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:13.903292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.903235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:13.903652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.903372 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:13.907507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:13.907484 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-6723f1305ac6\"" Oct 30 22:13:14.005807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.005772 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd96\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.005990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.005955 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.006107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.006089 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.006284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.006232 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.007139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.007095 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.007223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.007178 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.009496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.009470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.021516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.021476 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd96\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.229810 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:14.229778 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ca3f0c_35d0_40e2_9446_7deec6981b98.slice/crio-5d6f80f989194088e05921009b04ddecbb96feef95734206ace1aa5a97e1e6b6 WatchSource:0}: Error finding container 5d6f80f989194088e05921009b04ddecbb96feef95734206ace1aa5a97e1e6b6: Status 404 returned error can't find the container with id 5d6f80f989194088e05921009b04ddecbb96feef95734206ace1aa5a97e1e6b6 Oct 30 22:13:14.232724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.232693 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:14.279107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.279076 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:14.490351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.487876 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" event={"ID":"35ca3f0c-35d0-40e2-9446-7deec6981b98","Type":"ContainerStarted","Data":"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57"} Oct 30 22:13:14.490351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.487957 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" event={"ID":"35ca3f0c-35d0-40e2-9446-7deec6981b98","Type":"ContainerStarted","Data":"5d6f80f989194088e05921009b04ddecbb96feef95734206ace1aa5a97e1e6b6"} Oct 30 22:13:14.490351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.488024 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:14.512799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.512339 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" podStartSLOduration=1.5123232340000001 podStartE2EDuration="1.512323234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:14.511519094 +0000 UTC m=+2018.461474042" watchObservedRunningTime="2026-10-30 22:13:14.512323234 +0000 UTC m=+2018.462278193" Oct 30 22:13:14.580277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:14.580235 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:13:14.581086 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:14.581065 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb855833_6341_4475_bebe_9add129125bb.slice/crio-8d0ea1ebb3e0b86b27ebe36d4db302c8db0489b2abaddb3d8c398ac52d4b1e7e WatchSource:0}: Error finding container 8d0ea1ebb3e0b86b27ebe36d4db302c8db0489b2abaddb3d8c398ac52d4b1e7e: Status 404 returned error can't find the container with id 8d0ea1ebb3e0b86b27ebe36d4db302c8db0489b2abaddb3d8c398ac52d4b1e7e Oct 30 22:13:15.404561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:15.404526 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:15.499183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:15.499139 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" event={"ID":"bb855833-6341-4475-bebe-9add129125bb","Type":"ContainerStarted","Data":"80026d2cbd65420aea3a7b2e29a3981accef899ff145bc4b64fd3472ba122575"} Oct 30 22:13:15.499433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:15.499204 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" event={"ID":"bb855833-6341-4475-bebe-9add129125bb","Type":"ContainerStarted","Data":"8d0ea1ebb3e0b86b27ebe36d4db302c8db0489b2abaddb3d8c398ac52d4b1e7e"} Oct 30 22:13:15.499433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:15.499249 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:15.522686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:15.522629 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" podStartSLOduration=2.5226116960000002 podStartE2EDuration="2.522611696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:15.520184157 +0000 UTC m=+2019.470139114" watchObservedRunningTime="2026-10-30 22:13:15.522611696 +0000 UTC m=+2019.472566644" Oct 30 22:13:15.563049 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:15.563017 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-ac35153a2219: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:15.563049 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:15.563048 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:15.563351 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:15.563109 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:16.063089858 +0000 UTC m=+2020.013044803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : secret "agc-cred-ac35153a2219" not found Oct 30 22:13:15.563623 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:15.563601 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:15.563710 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:15.563650 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:16.063637258 +0000 UTC m=+2020.013592181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:16.084723 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:16.084693 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-ac35153a2219: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:16.084723 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:16.084727 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:16.085179 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:16.084779 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:17.084763192 +0000 UTC m=+2021.034718120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : secret "agc-cred-ac35153a2219" not found Oct 30 22:13:16.085179 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:16.084816 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:16.085179 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:16.084873 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:17.084860864 +0000 UTC m=+2021.034815788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:17.102676 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:17.102635 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:17.103147 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:17.102718 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:19.102698625 +0000 UTC m=+2023.052653566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:17.103147 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:17.102779 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-ac35153a2219: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:17.103147 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:17.102798 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:17.103147 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:17.102843 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:19.102830531 +0000 UTC m=+2023.052785454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : secret "agc-cred-ac35153a2219" not found Oct 30 22:13:18.100585 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:18.100547 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:13:19.145633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:19.144330 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-ac35153a2219: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:19.145633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:19.144360 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9: secret "agc-cred-ac35153a2219" not found Oct 30 22:13:19.145633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:19.144436 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:23.144412457 +0000 UTC m=+2027.094367400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : secret "agc-cred-ac35153a2219" not found Oct 30 22:13:19.145633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:19.144960 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:19.145633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:19.145019 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume podName:35ca3f0c-35d0-40e2-9446-7deec6981b98 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:23.145002326 +0000 UTC m=+2027.094957270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:13:21.510361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:21.510314 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" podUID="35ca3f0c-35d0-40e2-9446-7deec6981b98" containerName="payload-processing" containerID="cri-o://78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57" gracePeriod=25 Oct 30 22:13:21.839848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:21.839822 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:22.007087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\") pod \"35ca3f0c-35d0-40e2-9446-7deec6981b98\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " Oct 30 22:13:22.007421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007135 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\") pod \"35ca3f0c-35d0-40e2-9446-7deec6981b98\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " Oct 30 22:13:22.007421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007287 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\") pod \"35ca3f0c-35d0-40e2-9446-7deec6981b98\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " Oct 30 22:13:22.007542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\") pod \"35ca3f0c-35d0-40e2-9446-7deec6981b98\" (UID: \"35ca3f0c-35d0-40e2-9446-7deec6981b98\") " Oct 30 22:13:22.007608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007560 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume" pod "35ca3f0c-35d0-40e2-9446-7deec6981b98" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:22.007782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.007757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume" pod "35ca3f0c-35d0-40e2-9446-7deec6981b98" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:22.008768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.008744 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:22.008895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.008771 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ca3f0c-35d0-40e2-9446-7deec6981b98-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:22.009864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.009837 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials" pod "35ca3f0c-35d0-40e2-9446-7deec6981b98" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:22.009953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.009864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6" pod "35ca3f0c-35d0-40e2-9446-7deec6981b98" (UID: "35ca3f0c-35d0-40e2-9446-7deec6981b98"). InnerVolumeSpecName "kube-api-access-lqfw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:22.110699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.110615 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfw6\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-kube-api-access-lqfw6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:22.110699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.110644 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35ca3f0c-35d0-40e2-9446-7deec6981b98-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:22.592808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.592771 5036 generic.go:356] "Generic (PLEG): container finished" podID="35ca3f0c-35d0-40e2-9446-7deec6981b98" containerID="78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57" exitCode=0 Oct 30 22:13:22.594847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.592840 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" Oct 30 22:13:22.594847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.592876 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" event={"ID":"35ca3f0c-35d0-40e2-9446-7deec6981b98","Type":"ContainerDied","Data":"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57"} Oct 30 22:13:22.594847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.592983 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9" event={"ID":"35ca3f0c-35d0-40e2-9446-7deec6981b98","Type":"ContainerDied","Data":"5d6f80f989194088e05921009b04ddecbb96feef95734206ace1aa5a97e1e6b6"} Oct 30 22:13:22.594847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.593041 5036 scope.go:122] "RemoveContainer" containerID="78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57" Oct 30 22:13:22.623942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.623903 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:22.626246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.626228 5036 scope.go:122] "RemoveContainer" containerID="78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57" Oct 30 22:13:22.637164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.637130 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-69bffg7v9"] Oct 30 22:13:22.642754 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:22.642720 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57\": container with ID starting with 78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57 not found: ID does not exist" containerID="78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57" Oct 30 22:13:22.642877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:22.642764 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57"} err="failed to get container status \"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57\": rpc error: code = NotFound desc = could not find container \"78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57\": container with ID starting with 78a69300f8e965ddf9ebdeb9973203d74712150da7347a30f8efc6dcda9a4c57 not found: ID does not exist" Oct 30 22:13:24.191497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:24.191465 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ca3f0c-35d0-40e2-9446-7deec6981b98" path="/var/lib/kubelet/pods/35ca3f0c-35d0-40e2-9446-7deec6981b98/volumes" Oct 30 22:13:24.491824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:24.491738 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:13:24.589389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:24.589354 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:13:26.513656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:26.513622 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:13:26.575289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:26.575198 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:13:27.783348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.779499 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:13:27.783348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.780950 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35ca3f0c-35d0-40e2-9446-7deec6981b98" containerName="payload-processing" Oct 30 22:13:27.783348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.780977 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ca3f0c-35d0-40e2-9446-7deec6981b98" containerName="payload-processing" Oct 30 22:13:27.783348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.781211 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ca3f0c-35d0-40e2-9446-7deec6981b98" containerName="payload-processing" Oct 30 22:13:27.802238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.802209 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.803562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.802804 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:13:27.813552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.811356 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a5f192-openshift-default-dockercfg-829z5\"" Oct 30 22:13:27.873704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.873663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.873888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.873867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslmk\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.873974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.873929 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874127 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874178 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874444 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.874571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.874549 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976288 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976450 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976607 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976664 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslmk\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976906 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.976959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.976943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.977173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.977150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.977332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.977307 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.977410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.977380 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.977651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.977623 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.979037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.979006 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.979681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.979658 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.988561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.988534 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:27.988957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:27.988935 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslmk\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:28.153484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.153453 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:28.605372 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:28.605329 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a1df31_806c_4737_b116_2be2784a731b.slice/crio-0ca47b6b5a4982894a991772a0f2e48a16d03e706788780e8d3bcc1e3280ebcf WatchSource:0}: Error finding container 0ca47b6b5a4982894a991772a0f2e48a16d03e706788780e8d3bcc1e3280ebcf: Status 404 returned error can't find the container with id 0ca47b6b5a4982894a991772a0f2e48a16d03e706788780e8d3bcc1e3280ebcf Oct 30 22:13:28.607818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.607781 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:28.607935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.607875 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:28.607935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.607923 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:28.608744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.608600 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:13:28.701788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:28.690762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" event={"ID":"04a1df31-806c-4737-b116-2be2784a731b","Type":"ContainerStarted","Data":"0ca47b6b5a4982894a991772a0f2e48a16d03e706788780e8d3bcc1e3280ebcf"} Oct 30 22:13:29.592410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:29.590340 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" podUID="5233cbcb-e390-4a37-90dc-e022bd586c56" containerName="payload-processing" containerID="cri-o://bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9" gracePeriod=25 Oct 30 22:13:29.703380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:29.703332 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" event={"ID":"04a1df31-806c-4737-b116-2be2784a731b","Type":"ContainerStarted","Data":"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5"} Oct 30 22:13:29.743075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:29.742973 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podStartSLOduration=2.742952573 podStartE2EDuration="2.742952573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:29.737771043 +0000 UTC m=+2033.687725990" watchObservedRunningTime="2026-10-30 22:13:29.742952573 +0000 UTC m=+2033.692907522" Oct 30 22:13:29.998660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:29.998632 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:13:30.150947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.150905 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\") pod \"5233cbcb-e390-4a37-90dc-e022bd586c56\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " Oct 30 22:13:30.163949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.155581 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\") pod \"5233cbcb-e390-4a37-90dc-e022bd586c56\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " Oct 30 22:13:30.185292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.164384 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\") pod \"5233cbcb-e390-4a37-90dc-e022bd586c56\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " Oct 30 22:13:30.185776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.185742 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\") pod \"5233cbcb-e390-4a37-90dc-e022bd586c56\" (UID: \"5233cbcb-e390-4a37-90dc-e022bd586c56\") " Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.156580 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.159219 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.208202 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.173084 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume" pod "5233cbcb-e390-4a37-90dc-e022bd586c56" (UID: "5233cbcb-e390-4a37-90dc-e022bd586c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.180534 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume" pod "5233cbcb-e390-4a37-90dc-e022bd586c56" (UID: "5233cbcb-e390-4a37-90dc-e022bd586c56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.188146 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials" pod "5233cbcb-e390-4a37-90dc-e022bd586c56" (UID: "5233cbcb-e390-4a37-90dc-e022bd586c56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:30.211344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.191999 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97" pod "5233cbcb-e390-4a37-90dc-e022bd586c56" (UID: "5233cbcb-e390-4a37-90dc-e022bd586c56"). InnerVolumeSpecName "kube-api-access-p6p97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:30.288710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.288675 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.288710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.288705 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5233cbcb-e390-4a37-90dc-e022bd586c56-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.288710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.288717 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6p97\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-kube-api-access-p6p97\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.288943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.288726 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5233cbcb-e390-4a37-90dc-e022bd586c56-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.747967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.747915 5036 generic.go:356] "Generic (PLEG): container finished" podID="5233cbcb-e390-4a37-90dc-e022bd586c56" containerID="bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9" exitCode=0 Oct 30 22:13:30.747967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.747962 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" Oct 30 22:13:30.748552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.747983 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" event={"ID":"5233cbcb-e390-4a37-90dc-e022bd586c56","Type":"ContainerDied","Data":"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9"} Oct 30 22:13:30.748552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.748057 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq" event={"ID":"5233cbcb-e390-4a37-90dc-e022bd586c56","Type":"ContainerDied","Data":"cc67fd19903cd5a7ce5f37d01facf6dfae7d0d9766eb1ffe1fdabc28e25976bf"} Oct 30 22:13:30.748552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.748102 5036 scope.go:122] "RemoveContainer" containerID="bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9" Oct 30 22:13:30.763364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.763335 5036 scope.go:122] "RemoveContainer" containerID="bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9" Oct 30 22:13:30.763691 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:30.763663 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9\": container with ID starting with bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9 not found: ID does not exist" containerID="bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9" Oct 30 22:13:30.763809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.763701 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9"} err="failed to get container status \"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9\": rpc error: code = NotFound desc = could not find container \"bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9\": container with ID starting with bc5a2b24c7ca0b6b59d605f3fd1ffa9dbd3d904c75dab5a1ee5380c7df6cbde9 not found: ID does not exist" Oct 30 22:13:30.771126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.771100 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:13:30.776353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:30.776332 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cd7nvzq"] Oct 30 22:13:31.154498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.154457 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 22:13:31.154720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.154520 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 22:13:31.575826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.575725 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" podUID="f5b1cb46-c848-422e-bc00-54f0acb6a290" containerName="payload-processing" containerID="cri-o://bc5d527fbbcdb0139175318e924ffa0812049156b560fccb5c1d4a7f6bc0c48b" gracePeriod=25 Oct 30 22:13:31.767482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.767440 5036 generic.go:356] "Generic (PLEG): container finished" podID="f5b1cb46-c848-422e-bc00-54f0acb6a290" containerID="bc5d527fbbcdb0139175318e924ffa0812049156b560fccb5c1d4a7f6bc0c48b" exitCode=0 Oct 30 22:13:31.767960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.767713 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" event={"ID":"f5b1cb46-c848-422e-bc00-54f0acb6a290","Type":"ContainerDied","Data":"bc5d527fbbcdb0139175318e924ffa0812049156b560fccb5c1d4a7f6bc0c48b"} Oct 30 22:13:31.990881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:31.990846 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:13:32.064931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.064709 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\") pod \"f5b1cb46-c848-422e-bc00-54f0acb6a290\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " Oct 30 22:13:32.064931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.064930 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\") pod \"f5b1cb46-c848-422e-bc00-54f0acb6a290\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " Oct 30 22:13:32.065211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.065053 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume" pod "f5b1cb46-c848-422e-bc00-54f0acb6a290" (UID: "f5b1cb46-c848-422e-bc00-54f0acb6a290"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:32.065211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.065063 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\") pod \"f5b1cb46-c848-422e-bc00-54f0acb6a290\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " Oct 30 22:13:32.065211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.065124 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\") pod \"f5b1cb46-c848-422e-bc00-54f0acb6a290\" (UID: \"f5b1cb46-c848-422e-bc00-54f0acb6a290\") " Oct 30 22:13:32.066954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.066922 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume" pod "f5b1cb46-c848-422e-bc00-54f0acb6a290" (UID: "f5b1cb46-c848-422e-bc00-54f0acb6a290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:32.067100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.066965 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.069525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.069499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials" pod "f5b1cb46-c848-422e-bc00-54f0acb6a290" (UID: "f5b1cb46-c848-422e-bc00-54f0acb6a290"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.075302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.072802 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z" pod "f5b1cb46-c848-422e-bc00-54f0acb6a290" (UID: "f5b1cb46-c848-422e-bc00-54f0acb6a290"). InnerVolumeSpecName "kube-api-access-s4x2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.155686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.155648 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 22:13:32.155949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.155917 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 22:13:32.169180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.169147 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4x2z\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-kube-api-access-s4x2z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.169355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.169183 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1cb46-c848-422e-bc00-54f0acb6a290-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.169355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.169201 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1cb46-c848-422e-bc00-54f0acb6a290-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.190720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.190693 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5233cbcb-e390-4a37-90dc-e022bd586c56" path="/var/lib/kubelet/pods/5233cbcb-e390-4a37-90dc-e022bd586c56/volumes" Oct 30 22:13:32.783500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.783468 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" Oct 30 22:13:32.783500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.783489 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg" event={"ID":"f5b1cb46-c848-422e-bc00-54f0acb6a290","Type":"ContainerDied","Data":"87934ae611fdc1d9c2fdc5d807720ca0c43bbaaa8e74d77b7ab4070532534115"} Oct 30 22:13:32.783980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.783552 5036 scope.go:122] "RemoveContainer" containerID="bc5d527fbbcdb0139175318e924ffa0812049156b560fccb5c1d4a7f6bc0c48b" Oct 30 22:13:32.803376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.803348 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:13:32.809153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:32.809131 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-64846rgqg"] Oct 30 22:13:33.154397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:33.154360 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 22:13:33.154611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:33.154443 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 22:13:34.154157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:34.154108 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 22:13:34.154668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:34.154173 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 22:13:34.191522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:34.191492 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b1cb46-c848-422e-bc00-54f0acb6a290" path="/var/lib/kubelet/pods/f5b1cb46-c848-422e-bc00-54f0acb6a290/volumes" Oct 30 22:13:35.168976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:35.168944 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:35.169823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:35.169793 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:35.171121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:35.171100 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:13:46.050318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:46.034638 5036 scope.go:122] "RemoveContainer" containerID="bcc82fe8b20c3232deec29da3f98a0ef95da0596dc0951673b35beb8355daa81" Oct 30 22:13:46.161913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:46.161873 5036 scope.go:122] "RemoveContainer" containerID="76cf534339da000b1a2e7dd76daba93615dcef3118adc8b0996a41cd9d811551" Oct 30 22:13:46.180566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:46.180539 5036 scope.go:122] "RemoveContainer" containerID="ef1c023c2e47dbbdb8a5972e4cef8ebb3d0e3c4bba80cb7511b2627f6b0a5f77" Oct 30 22:13:47.276748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.276709 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:13:47.324817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.324785 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f5b1cb46-c848-422e-bc00-54f0acb6a290" containerName="payload-processing" Oct 30 22:13:47.325191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.325169 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b1cb46-c848-422e-bc00-54f0acb6a290" containerName="payload-processing" Oct 30 22:13:47.325441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.325425 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5233cbcb-e390-4a37-90dc-e022bd586c56" containerName="payload-processing" Oct 30 22:13:47.325570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.325546 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5233cbcb-e390-4a37-90dc-e022bd586c56" containerName="payload-processing" Oct 30 22:13:47.326037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.326018 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b1cb46-c848-422e-bc00-54f0acb6a290" containerName="payload-processing" Oct 30 22:13:47.326182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.326170 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5233cbcb-e390-4a37-90dc-e022bd586c56" containerName="payload-processing" Oct 30 22:13:47.393288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.387616 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.467629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.412683 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-4b132b019164\"" Oct 30 22:13:47.615343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.611610 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntzk\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.615343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.611751 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.615343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.611908 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.615343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.611952 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.648296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.642483 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:13:47.725289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.713430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.725289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.713498 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.725289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.713738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntzk\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.725289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.714185 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.742291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.731324 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.742291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.736431 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.742291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.738484 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:47.752458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:47.751762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntzk\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:48.034293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:48.028587 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:48.167463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:48.163342 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:13:48.380454 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:48.380424 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a463b7_e764_4f43_be0c_6d1903f6f522.slice/crio-ee40d5c9c86e818cfe02faa84f844454366f897da4bed218b12ee7c4776590f8 WatchSource:0}: Error finding container ee40d5c9c86e818cfe02faa84f844454366f897da4bed218b12ee7c4776590f8: Status 404 returned error can't find the container with id ee40d5c9c86e818cfe02faa84f844454366f897da4bed218b12ee7c4776590f8 Oct 30 22:13:48.383303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:48.383252 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:13:49.245512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.244739 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" event={"ID":"45a463b7-e764-4f43-be0c-6d1903f6f522","Type":"ContainerStarted","Data":"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3"} Oct 30 22:13:49.245512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.244811 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" event={"ID":"45a463b7-e764-4f43-be0c-6d1903f6f522","Type":"ContainerStarted","Data":"ee40d5c9c86e818cfe02faa84f844454366f897da4bed218b12ee7c4776590f8"} Oct 30 22:13:49.245512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.245052 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:13:49.280967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.280901 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" podStartSLOduration=2.280886487 podStartE2EDuration="2.280886487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.279768766 +0000 UTC m=+2053.229723714" watchObservedRunningTime="2026-10-30 22:13:49.280886487 +0000 UTC m=+2053.230841444" Oct 30 22:13:49.697801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.697770 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:13:49.703027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.703008 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.747495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.747459 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a1ea1e\"/\"agc-cred-1be6c03b8107\"" Oct 30 22:13:49.747660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.747589 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:13:49.868023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.867985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdm7n\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.868211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.868054 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.868211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.868202 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.868404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.868390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.970111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.970015 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.970322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.970158 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.970322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.970258 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdm7n\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.970322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.970303 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.971015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.970987 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.971015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.971007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.973085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.973063 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:49.995927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:49.995901 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdm7n\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:50.039289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:50.036763 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:50.366184 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:50.366126 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b08ce44_f91f_42d0_bf09_1728afe32a83.slice/crio-f151cd7605b305d908fa7d0cd1179fadd4fa60f0d611c2c2c7482e45144fad22 WatchSource:0}: Error finding container f151cd7605b305d908fa7d0cd1179fadd4fa60f0d611c2c2c7482e45144fad22: Status 404 returned error can't find the container with id f151cd7605b305d908fa7d0cd1179fadd4fa60f0d611c2c2c7482e45144fad22 Oct 30 22:13:50.366385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:50.366310 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:13:51.274950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:51.274910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" event={"ID":"1b08ce44-f91f-42d0-bf09-1728afe32a83","Type":"ContainerStarted","Data":"02b934cef5ca896f905f36300f4a530261387b9db913ddbe5ec6544fa33ba13b"} Oct 30 22:13:51.275399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:51.274969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" event={"ID":"1b08ce44-f91f-42d0-bf09-1728afe32a83","Type":"ContainerStarted","Data":"f151cd7605b305d908fa7d0cd1179fadd4fa60f0d611c2c2c7482e45144fad22"} Oct 30 22:13:51.275399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:51.275037 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:51.321589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:51.321504 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" podStartSLOduration=2.321481663 podStartE2EDuration="2.321481663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:51.303373931 +0000 UTC m=+2055.253328894" watchObservedRunningTime="2026-10-30 22:13:51.321481663 +0000 UTC m=+2055.271436611" Oct 30 22:13:54.199231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.199198 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr"] Oct 30 22:13:54.204521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.204501 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.210418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.210378 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:54.210575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.210537 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:13:54.210876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.210718 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:54.210876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.210842 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:54.215647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.215312 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr"] Oct 30 22:13:54.290645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.290611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.290887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.290866 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfjj\" (UniqueName: \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.290994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.290975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.392386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.392346 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.392607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.392492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.392689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.392662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfjj\" (UniqueName: \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.393183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.393153 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.395491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.395415 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.425697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.425660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfjj\" (UniqueName: \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\") pod \"maas-api-revoke-sub-bece7096-dq6zr\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:54.530193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:54.528785 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:55.124310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.123016 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr"] Oct 30 22:13:55.385423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.380589 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:13:55.438032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.437959 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:13:55.481286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.479359 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:13:55.481286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.479459 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:55.564978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:55.564937 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" event={"ID":"5508dbea-bc66-4991-a689-f1baaabd5e6a","Type":"ContainerStarted","Data":"a6b5cf98cf547d78310b879e51a6d67e0bcca8def4c012a8e1b3a1f0764efd41"} Oct 30 22:13:56.332614 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:13:56.332576 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5508dbea_bc66_4991_a689_f1baaabd5e6a.slice/crio-51c3844ceae43b11acaf1c48c18887e6324631bdd6a18c03c1ec0e7a2e2373f2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:13:56.577302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:56.577181 5036 generic.go:356] "Generic (PLEG): container finished" podID="5508dbea-bc66-4991-a689-f1baaabd5e6a" containerID="51c3844ceae43b11acaf1c48c18887e6324631bdd6a18c03c1ec0e7a2e2373f2" exitCode=0 Oct 30 22:13:56.577302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:56.577284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" event={"ID":"5508dbea-bc66-4991-a689-f1baaabd5e6a","Type":"ContainerDied","Data":"51c3844ceae43b11acaf1c48c18887e6324631bdd6a18c03c1ec0e7a2e2373f2"} Oct 30 22:13:57.486921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.486851 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw"] Oct 30 22:13:57.491756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.491735 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.503461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.503437 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw"] Oct 30 22:13:57.591167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.591076 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerID="12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b" exitCode=0 Oct 30 22:13:57.591167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.591144 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerDied","Data":"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b"} Oct 30 22:13:57.591716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.591695 5036 scope.go:122] "RemoveContainer" containerID="12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b" Oct 30 22:13:57.594002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.593965 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:57.594709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.594052 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:57.594709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.594099 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:57.675574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.673016 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.681810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.681747 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ws2k\" (UniqueName: \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.682232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.682207 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.784458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.784397 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ws2k\" (UniqueName: \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.784749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.784728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.784937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.784923 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.788328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.787597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.794941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.794901 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.799950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.799896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ws2k\" (UniqueName: \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\") pod \"maas-api-revoke-sub-57e705b3-qjssw\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.847424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.827875 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:13:57.986893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:57.986869 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:58.091875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.091834 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\" (UniqueName: \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\") pod \"5508dbea-bc66-4991-a689-f1baaabd5e6a\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " Oct 30 22:13:58.092250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.092227 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\") pod \"5508dbea-bc66-4991-a689-f1baaabd5e6a\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " Oct 30 22:13:58.092366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.092333 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\") pod \"5508dbea-bc66-4991-a689-f1baaabd5e6a\" (UID: \"5508dbea-bc66-4991-a689-f1baaabd5e6a\") " Oct 30 22:13:58.093321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.093246 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca" pod "5508dbea-bc66-4991-a689-f1baaabd5e6a" (UID: "5508dbea-bc66-4991-a689-f1baaabd5e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:58.094205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.094175 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:58.096051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.096014 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj" pod "5508dbea-bc66-4991-a689-f1baaabd5e6a" (UID: "5508dbea-bc66-4991-a689-f1baaabd5e6a"). InnerVolumeSpecName "kube-api-access-xdfjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:58.096162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.096130 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca" pod "5508dbea-bc66-4991-a689-f1baaabd5e6a" (UID: "5508dbea-bc66-4991-a689-f1baaabd5e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:58.195503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.195465 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5508dbea-bc66-4991-a689-f1baaabd5e6a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:58.195503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.195498 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfjj\" (UniqueName: \"kubernetes.io/projected/5508dbea-bc66-4991-a689-f1baaabd5e6a-kube-api-access-xdfjj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:58.206361 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:13:58.206337 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74a80ea_c6f9_4e55_8729_3f4c434ed52a.slice/crio-e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a WatchSource:0}: Error finding container e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a: Status 404 returned error can't find the container with id e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a Oct 30 22:13:58.209514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.209488 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw"] Oct 30 22:13:58.618324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.616456 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" event={"ID":"a74a80ea-c6f9-4e55-8729-3f4c434ed52a","Type":"ContainerStarted","Data":"e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a"} Oct 30 22:13:58.618324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.617999 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" event={"ID":"5508dbea-bc66-4991-a689-f1baaabd5e6a","Type":"ContainerDied","Data":"a6b5cf98cf547d78310b879e51a6d67e0bcca8def4c012a8e1b3a1f0764efd41"} Oct 30 22:13:58.618324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.618034 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b5cf98cf547d78310b879e51a6d67e0bcca8def4c012a8e1b3a1f0764efd41" Oct 30 22:13:58.618324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.618091 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr" Oct 30 22:13:58.620630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.620604 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerStarted","Data":"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace"} Oct 30 22:13:58.836059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.835969 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:13:58.836487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.836463 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:58.836684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:58.836648 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:13:59.662650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:59.662618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" event={"ID":"a74a80ea-c6f9-4e55-8729-3f4c434ed52a","Type":"ContainerStarted","Data":"e9f8edb6072dcc71865065416899f36252be99f24b1677bded547a294212de18"} Oct 30 22:13:59.680966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:59.680916 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" podStartSLOduration=2.680898616 podStartE2EDuration="2.680898616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:59.679423136 +0000 UTC m=+2063.629378081" watchObservedRunningTime="2026-10-30 22:13:59.680898616 +0000 UTC m=+2063.630853562" Oct 30 22:13:59.759795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:59.759759 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:13:59.838292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:59.837637 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:59.838292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:13:59.837697 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:14:00.265719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.265691 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:14:00.359623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.359587 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:14:00.407598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.407553 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" podUID="1b08ce44-f91f-42d0-bf09-1728afe32a83" containerName="payload-processing" containerID="cri-o://02b934cef5ca896f905f36300f4a530261387b9db913ddbe5ec6544fa33ba13b" gracePeriod=25 Oct 30 22:14:00.678570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.678531 5036 generic.go:356] "Generic (PLEG): container finished" podID="a74a80ea-c6f9-4e55-8729-3f4c434ed52a" containerID="e9f8edb6072dcc71865065416899f36252be99f24b1677bded547a294212de18" exitCode=0 Oct 30 22:14:00.679212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.678612 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" event={"ID":"a74a80ea-c6f9-4e55-8729-3f4c434ed52a","Type":"ContainerDied","Data":"e9f8edb6072dcc71865065416899f36252be99f24b1677bded547a294212de18"} Oct 30 22:14:00.681024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.680999 5036 generic.go:356] "Generic (PLEG): container finished" podID="1b08ce44-f91f-42d0-bf09-1728afe32a83" containerID="02b934cef5ca896f905f36300f4a530261387b9db913ddbe5ec6544fa33ba13b" exitCode=0 Oct 30 22:14:00.681217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.681067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" event={"ID":"1b08ce44-f91f-42d0-bf09-1728afe32a83","Type":"ContainerDied","Data":"02b934cef5ca896f905f36300f4a530261387b9db913ddbe5ec6544fa33ba13b"} Oct 30 22:14:00.861510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.859193 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:14:00.861510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.859251 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:14:00.960459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:00.960430 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.063164 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\") pod \"1b08ce44-f91f-42d0-bf09-1728afe32a83\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.063224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\") pod \"1b08ce44-f91f-42d0-bf09-1728afe32a83\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.063401 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\") pod \"1b08ce44-f91f-42d0-bf09-1728afe32a83\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.063536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\") pod \"1b08ce44-f91f-42d0-bf09-1728afe32a83\" (UID: \"1b08ce44-f91f-42d0-bf09-1728afe32a83\") " Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.064839 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume" pod "1b08ce44-f91f-42d0-bf09-1728afe32a83" (UID: "1b08ce44-f91f-42d0-bf09-1728afe32a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:01.066324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.065178 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume" pod "1b08ce44-f91f-42d0-bf09-1728afe32a83" (UID: "1b08ce44-f91f-42d0-bf09-1728afe32a83"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:01.067451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.067419 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials" pod "1b08ce44-f91f-42d0-bf09-1728afe32a83" (UID: "1b08ce44-f91f-42d0-bf09-1728afe32a83"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:01.068941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.068911 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n" pod "1b08ce44-f91f-42d0-bf09-1728afe32a83" (UID: "1b08ce44-f91f-42d0-bf09-1728afe32a83"). InnerVolumeSpecName "kube-api-access-wdm7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:01.166511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.166386 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.166511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.166423 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdm7n\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-kube-api-access-wdm7n\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.166511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.166438 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b08ce44-f91f-42d0-bf09-1728afe32a83-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.166511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.166481 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b08ce44-f91f-42d0-bf09-1728afe32a83-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.691847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.691815 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" Oct 30 22:14:01.691847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.691824 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb" event={"ID":"1b08ce44-f91f-42d0-bf09-1728afe32a83","Type":"ContainerDied","Data":"f151cd7605b305d908fa7d0cd1179fadd4fa60f0d611c2c2c7482e45144fad22"} Oct 30 22:14:01.692407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.691893 5036 scope.go:122] "RemoveContainer" containerID="02b934cef5ca896f905f36300f4a530261387b9db913ddbe5ec6544fa33ba13b" Oct 30 22:14:01.751049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.751011 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:14:01.760416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.760377 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-84d9q8nrb"] Oct 30 22:14:01.863195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.863154 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:14:01.863413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:01.863218 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:14:02.107897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.107867 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:14:02.193708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.193678 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b08ce44-f91f-42d0-bf09-1728afe32a83" path="/var/lib/kubelet/pods/1b08ce44-f91f-42d0-bf09-1728afe32a83/volumes" Oct 30 22:14:02.246864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.246814 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\" (UniqueName: \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\") pod \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " Oct 30 22:14:02.247060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.246944 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\") pod \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " Oct 30 22:14:02.247060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.247032 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\") pod \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\" (UID: \"a74a80ea-c6f9-4e55-8729-3f4c434ed52a\") " Oct 30 22:14:02.247941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.247879 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca" pod "a74a80ea-c6f9-4e55-8729-3f4c434ed52a" (UID: "a74a80ea-c6f9-4e55-8729-3f4c434ed52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:02.248737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.248717 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:02.252028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.252006 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca" pod "a74a80ea-c6f9-4e55-8729-3f4c434ed52a" (UID: "a74a80ea-c6f9-4e55-8729-3f4c434ed52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:02.252915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.252882 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k" pod "a74a80ea-c6f9-4e55-8729-3f4c434ed52a" (UID: "a74a80ea-c6f9-4e55-8729-3f4c434ed52a"). InnerVolumeSpecName "kube-api-access-4ws2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:02.350715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.350676 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ws2k\" (UniqueName: \"kubernetes.io/projected/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-kube-api-access-4ws2k\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:02.350715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.350709 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74a80ea-c6f9-4e55-8729-3f4c434ed52a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:02.706043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.705995 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" event={"ID":"a74a80ea-c6f9-4e55-8729-3f4c434ed52a","Type":"ContainerDied","Data":"e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a"} Oct 30 22:14:02.706510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.706067 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38d6558c1019e4faf9d433d2bb56ffcc36be124cec391d810c448cc9b7aea6a" Oct 30 22:14:02.706510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.706126 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw" Oct 30 22:14:02.839294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.838703 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:14:02.839294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:02.838770 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:14:03.839491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:03.839452 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 22:14:03.840119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:03.840081 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 22:14:04.760931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:04.760885 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" podUID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" containerName="payload-processing" containerID="cri-o://dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252" gracePeriod=25 Oct 30 22:14:04.844678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:04.844643 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:14:04.845735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:04.845195 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:14:04.846611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:04.846586 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:14:05.155272 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.155240 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:14:05.231760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.230927 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\") pod \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " Oct 30 22:14:05.231760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.231062 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\") pod \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " Oct 30 22:14:05.231760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.231110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\") pod \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " Oct 30 22:14:05.231760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.231343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\") pod \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\" (UID: \"2bb1bdc8-564f-4f19-814c-ecfec7287dee\") " Oct 30 22:14:05.232975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.232941 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume" pod "2bb1bdc8-564f-4f19-814c-ecfec7287dee" (UID: "2bb1bdc8-564f-4f19-814c-ecfec7287dee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:05.233388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.233364 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume" pod "2bb1bdc8-564f-4f19-814c-ecfec7287dee" (UID: "2bb1bdc8-564f-4f19-814c-ecfec7287dee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:05.246421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.246391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials" pod "2bb1bdc8-564f-4f19-814c-ecfec7287dee" (UID: "2bb1bdc8-564f-4f19-814c-ecfec7287dee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:05.250411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.250301 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh" pod "2bb1bdc8-564f-4f19-814c-ecfec7287dee" (UID: "2bb1bdc8-564f-4f19-814c-ecfec7287dee"). InnerVolumeSpecName "kube-api-access-tzrgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:05.372368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.369353 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" podUID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" containerName="payload-processing" containerID="cri-o://d2f5af6a6d8ea39e41f325ed2752bbc042c01935a56baec077e1af5ee06f8867" gracePeriod=25 Oct 30 22:14:05.386695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.386659 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:05.394492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.394452 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb1bdc8-564f-4f19-814c-ecfec7287dee-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:05.394818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.394801 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrgh\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-kube-api-access-tzrgh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:05.394964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.394950 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bb1bdc8-564f-4f19-814c-ecfec7287dee-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:05.449326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.449163 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450448 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1b08ce44-f91f-42d0-bf09-1728afe32a83" containerName="payload-processing" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450474 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b08ce44-f91f-42d0-bf09-1728afe32a83" containerName="payload-processing" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450496 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" containerName="payload-processing" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450504 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" containerName="payload-processing" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450554 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5508dbea-bc66-4991-a689-f1baaabd5e6a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450561 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5508dbea-bc66-4991-a689-f1baaabd5e6a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450571 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a74a80ea-c6f9-4e55-8729-3f4c434ed52a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450579 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a80ea-c6f9-4e55-8729-3f4c434ed52a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450767 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" containerName="payload-processing" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450787 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5508dbea-bc66-4991-a689-f1baaabd5e6a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a80ea-c6f9-4e55-8729-3f4c434ed52a" containerName="revoke-keys" Oct 30 22:14:05.454338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.450818 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b08ce44-f91f-42d0-bf09-1728afe32a83" containerName="payload-processing" Oct 30 22:14:05.476296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.472319 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.498299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.498071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpsk\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.498459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.498368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.498575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.498472 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.498575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.498523 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.566195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.566161 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:05.600155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.600120 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpsk\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.600605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.600325 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.600605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.600405 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.600605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.600453 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.601411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.601384 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.602448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.602417 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.604893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.604869 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.619874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.619848 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpsk\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.779532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.777688 5036 generic.go:356] "Generic (PLEG): container finished" podID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" containerID="dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252" exitCode=0 Oct 30 22:14:05.779532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.777960 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" Oct 30 22:14:05.779532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.778128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" event={"ID":"2bb1bdc8-564f-4f19-814c-ecfec7287dee","Type":"ContainerDied","Data":"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252"} Oct 30 22:14:05.779532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.778196 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb" event={"ID":"2bb1bdc8-564f-4f19-814c-ecfec7287dee","Type":"ContainerDied","Data":"b1c31dfdf28d56241353060902b335e8043efa7ba17f31178bb56c60831f0f1f"} Oct 30 22:14:05.779532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.778226 5036 scope.go:122] "RemoveContainer" containerID="dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252" Oct 30 22:14:05.780508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.780481 5036 generic.go:356] "Generic (PLEG): container finished" podID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" containerID="d2f5af6a6d8ea39e41f325ed2752bbc042c01935a56baec077e1af5ee06f8867" exitCode=0 Oct 30 22:14:05.781365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.780664 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" event={"ID":"abf61fc1-d7a7-4c4a-9534-c849b8daf88f","Type":"ContainerDied","Data":"d2f5af6a6d8ea39e41f325ed2752bbc042c01935a56baec077e1af5ee06f8867"} Oct 30 22:14:05.791967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.791942 5036 scope.go:122] "RemoveContainer" containerID="dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252" Oct 30 22:14:05.792340 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:05.792311 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252\": container with ID starting with dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252 not found: ID does not exist" containerID="dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252" Oct 30 22:14:05.792429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.792352 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252"} err="failed to get container status \"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252\": rpc error: code = NotFound desc = could not find container \"dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252\": container with ID starting with dac0dca1502a5b152b60c69ecafb50a29bfe78845459aee472354c1bd847a252 not found: ID does not exist" Oct 30 22:14:05.803278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.803234 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:14:05.810656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.810631 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:14:05.888290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.887481 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:05.911755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.909892 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\") pod \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " Oct 30 22:14:05.911755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.910017 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\") pod \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " Oct 30 22:14:05.911755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.910077 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\") pod \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " Oct 30 22:14:05.911755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.910228 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\") pod \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\" (UID: \"abf61fc1-d7a7-4c4a-9534-c849b8daf88f\") " Oct 30 22:14:05.922667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.922614 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume" pod "abf61fc1-d7a7-4c4a-9534-c849b8daf88f" (UID: "abf61fc1-d7a7-4c4a-9534-c849b8daf88f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:05.923405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.923378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume" pod "abf61fc1-d7a7-4c4a-9534-c849b8daf88f" (UID: "abf61fc1-d7a7-4c4a-9534-c849b8daf88f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:05.948505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.941777 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials" pod "abf61fc1-d7a7-4c4a-9534-c849b8daf88f" (UID: "abf61fc1-d7a7-4c4a-9534-c849b8daf88f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:05.948505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.941935 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5jz7xb"] Oct 30 22:14:05.952513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.952483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9" pod "abf61fc1-d7a7-4c4a-9534-c849b8daf88f" (UID: "abf61fc1-d7a7-4c4a-9534-c849b8daf88f"). InnerVolumeSpecName "kube-api-access-ps6p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:05.952710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.952658 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq"] Oct 30 22:14:05.955536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.955516 5036 cpu_manager.go:414] "containerMap: removing container" podUID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" containerName="payload-processing" Oct 30 22:14:05.955675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.955661 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" containerName="payload-processing" Oct 30 22:14:05.956075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.956059 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" containerName="payload-processing" Oct 30 22:14:05.973343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.973309 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:05.977934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.977912 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:05.978190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.978172 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:14:05.978551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.978534 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:05.978718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:05.978705 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:06.020002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.019156 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.020002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.019189 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.020002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.019203 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps6p9\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-kube-api-access-ps6p9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.020002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.019215 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abf61fc1-d7a7-4c4a-9534-c849b8daf88f-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.033763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.033669 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq"] Oct 30 22:14:06.129307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.126195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cpf\" (UniqueName: \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.129307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.126260 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.129307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.126488 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.234356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.233780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.236034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.235820 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cpf\" (UniqueName: \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.237547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.237207 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.238215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.234308 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb1bdc8-564f-4f19-814c-ecfec7287dee" path="/var/lib/kubelet/pods/2bb1bdc8-564f-4f19-814c-ecfec7287dee/volumes" Oct 30 22:14:06.243831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.243307 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.243831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.235230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.252156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.252108 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cpf\" (UniqueName: \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\") pod \"maas-api-revoke-sub-4ac08523-c9phq\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.390469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.390435 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:06.538387 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:06.538351 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbcd5aa_b2db_489e_9110_ee00c86b8e57.slice/crio-b26bdcfb6033e018085af79c5489d84933cbe5f948291f551aab1ee709097afe WatchSource:0}: Error finding container b26bdcfb6033e018085af79c5489d84933cbe5f948291f551aab1ee709097afe: Status 404 returned error can't find the container with id b26bdcfb6033e018085af79c5489d84933cbe5f948291f551aab1ee709097afe Oct 30 22:14:06.540326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.540239 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:06.723320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.717648 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq"] Oct 30 22:14:06.791160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.791106 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" Oct 30 22:14:06.791342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.791105 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r" event={"ID":"abf61fc1-d7a7-4c4a-9534-c849b8daf88f","Type":"ContainerDied","Data":"77f0e3d1eb806b4a3d3bf9dc403adb180e8da966c9e5fda29a5317e48e85d8dc"} Oct 30 22:14:06.791342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.791312 5036 scope.go:122] "RemoveContainer" containerID="d2f5af6a6d8ea39e41f325ed2752bbc042c01935a56baec077e1af5ee06f8867" Oct 30 22:14:06.794608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.793704 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" event={"ID":"8dbcd5aa-b2db-489e-9110-ee00c86b8e57","Type":"ContainerStarted","Data":"75a3b68aa740cebe30c1638b6d5788aed0c1d7cfa02c5f014cdd8eab08998523"} Oct 30 22:14:06.794608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.793759 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" event={"ID":"8dbcd5aa-b2db-489e-9110-ee00c86b8e57","Type":"ContainerStarted","Data":"b26bdcfb6033e018085af79c5489d84933cbe5f948291f551aab1ee709097afe"} Oct 30 22:14:06.794608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.794548 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:06.795523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.795502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" event={"ID":"8a210acb-cbd9-4134-be02-a1ce8bc48152","Type":"ContainerStarted","Data":"b203daceda5e204713cdfebd5c9db0a85bee0fb21285f009134956a7137234ec"} Oct 30 22:14:06.824590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.824537 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" podStartSLOduration=1.824502502 podStartE2EDuration="1.824502502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:06.816095054 +0000 UTC m=+2070.766049999" watchObservedRunningTime="2026-10-30 22:14:06.824502502 +0000 UTC m=+2070.774457449" Oct 30 22:14:06.837258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.837232 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:14:06.842329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:06.842304 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77584nn5r"] Oct 30 22:14:07.809430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:07.809395 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a210acb-cbd9-4134-be02-a1ce8bc48152" containerID="30d57e16e8d6333fa8692eb3a6fe5430eb506e0fc5455a18abba11711a38645e" exitCode=0 Oct 30 22:14:07.809892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:07.809456 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" event={"ID":"8a210acb-cbd9-4134-be02-a1ce8bc48152","Type":"ContainerDied","Data":"30d57e16e8d6333fa8692eb3a6fe5430eb506e0fc5455a18abba11711a38645e"} Oct 30 22:14:08.195656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.195614 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf61fc1-d7a7-4c4a-9534-c849b8daf88f" path="/var/lib/kubelet/pods/abf61fc1-d7a7-4c4a-9534-c849b8daf88f/volumes" Oct 30 22:14:08.727128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.727092 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:08.732505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.732480 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.743199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.743160 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:08.848341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.835912 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.848341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.836076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.848341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.836160 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.848341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.836213 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwc9\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.951288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.945589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.951288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.945813 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.951288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.945919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.951288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.945989 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwc9\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.990006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.989900 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:08.995303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:08.992049 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:09.088083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.088046 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwc9\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:09.103895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.103860 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:09.109856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.109826 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7"] Oct 30 22:14:09.139286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.139238 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.141729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.141091 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7"] Oct 30 22:14:09.252416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.252338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltql\" (UniqueName: \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.252577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.252411 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.252577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.252458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.351301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.351258 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:09.354155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.354128 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltql\" (UniqueName: \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.354257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.354183 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.354257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.354219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.355180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.355053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.356936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.356912 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.368544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.368508 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltql\" (UniqueName: \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\") pod \"maas-api-revoke-sub-9185c28b-pxdz7\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.434500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.434465 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:09.472128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.471798 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:09.567664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.567398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\") pod \"8a210acb-cbd9-4134-be02-a1ce8bc48152\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " Oct 30 22:14:09.567664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.567484 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\") pod \"8a210acb-cbd9-4134-be02-a1ce8bc48152\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " Oct 30 22:14:09.567922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.567745 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\" (UniqueName: \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\") pod \"8a210acb-cbd9-4134-be02-a1ce8bc48152\" (UID: \"8a210acb-cbd9-4134-be02-a1ce8bc48152\") " Oct 30 22:14:09.569466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.569426 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca" pod "8a210acb-cbd9-4134-be02-a1ce8bc48152" (UID: "8a210acb-cbd9-4134-be02-a1ce8bc48152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:09.571802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.571771 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca" pod "8a210acb-cbd9-4134-be02-a1ce8bc48152" (UID: "8a210acb-cbd9-4134-be02-a1ce8bc48152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:09.576228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.576177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf" pod "8a210acb-cbd9-4134-be02-a1ce8bc48152" (UID: "8a210acb-cbd9-4134-be02-a1ce8bc48152"). InnerVolumeSpecName "kube-api-access-g4cpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:09.670735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.670699 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cpf\" (UniqueName: \"kubernetes.io/projected/8a210acb-cbd9-4134-be02-a1ce8bc48152-kube-api-access-g4cpf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.670735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.670738 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.670946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.670754 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a210acb-cbd9-4134-be02-a1ce8bc48152-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.724097 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:09.724069 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b113ad2_e384_40f2_98c6_c09539b4b82f.slice/crio-224994271d931f94ad31b2f40ab17a75073a6c1487e054903674acc57e75a74c WatchSource:0}: Error finding container 224994271d931f94ad31b2f40ab17a75073a6c1487e054903674acc57e75a74c: Status 404 returned error can't find the container with id 224994271d931f94ad31b2f40ab17a75073a6c1487e054903674acc57e75a74c Oct 30 22:14:09.727501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.727380 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:09.793913 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:09.793880 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70506294_9e8f_4aee_9870_1961472ca651.slice/crio-2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa WatchSource:0}: Error finding container 2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa: Status 404 returned error can't find the container with id 2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa Oct 30 22:14:09.795409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:09.795381 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7"] Oct 30 22:14:10.090958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.090906 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" event={"ID":"70506294-9e8f-4aee-9870-1961472ca651","Type":"ContainerStarted","Data":"2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa"} Oct 30 22:14:10.093586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.093554 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" event={"ID":"4b113ad2-e384-40f2-98c6-c09539b4b82f","Type":"ContainerStarted","Data":"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f"} Oct 30 22:14:10.093737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.093620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" event={"ID":"4b113ad2-e384-40f2-98c6-c09539b4b82f","Type":"ContainerStarted","Data":"224994271d931f94ad31b2f40ab17a75073a6c1487e054903674acc57e75a74c"} Oct 30 22:14:10.094566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.094532 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:10.096737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.096715 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" event={"ID":"8a210acb-cbd9-4134-be02-a1ce8bc48152","Type":"ContainerDied","Data":"b203daceda5e204713cdfebd5c9db0a85bee0fb21285f009134956a7137234ec"} Oct 30 22:14:10.096900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.096883 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b203daceda5e204713cdfebd5c9db0a85bee0fb21285f009134956a7137234ec" Oct 30 22:14:10.097114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.097097 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq" Oct 30 22:14:10.114981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:10.114936 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" podStartSLOduration=2.114922744 podStartE2EDuration="2.114922744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:10.112941047 +0000 UTC m=+2074.062895996" watchObservedRunningTime="2026-10-30 22:14:10.114922744 +0000 UTC m=+2074.064877699" Oct 30 22:14:11.109352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:11.109317 5036 generic.go:356] "Generic (PLEG): container finished" podID="70506294-9e8f-4aee-9870-1961472ca651" containerID="1476dbbc48e8eda240f3114516643de2913822ab4d8a412bde3e25c6f9768dd0" exitCode=0 Oct 30 22:14:11.109860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:11.109401 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" event={"ID":"70506294-9e8f-4aee-9870-1961472ca651","Type":"ContainerDied","Data":"1476dbbc48e8eda240f3114516643de2913822ab4d8a412bde3e25c6f9768dd0"} Oct 30 22:14:12.412347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.412321 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:12.449008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.448973 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\" (UniqueName: \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\") pod \"70506294-9e8f-4aee-9870-1961472ca651\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " Oct 30 22:14:12.449218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.449126 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\") pod \"70506294-9e8f-4aee-9870-1961472ca651\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " Oct 30 22:14:12.449218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.449210 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\") pod \"70506294-9e8f-4aee-9870-1961472ca651\" (UID: \"70506294-9e8f-4aee-9870-1961472ca651\") " Oct 30 22:14:12.450312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.450256 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca" pod "70506294-9e8f-4aee-9870-1961472ca651" (UID: "70506294-9e8f-4aee-9870-1961472ca651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:12.452524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.452233 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca" pod "70506294-9e8f-4aee-9870-1961472ca651" (UID: "70506294-9e8f-4aee-9870-1961472ca651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:12.455433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.452962 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql" pod "70506294-9e8f-4aee-9870-1961472ca651" (UID: "70506294-9e8f-4aee-9870-1961472ca651"). InnerVolumeSpecName "kube-api-access-sltql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:12.551880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.551833 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sltql\" (UniqueName: \"kubernetes.io/projected/70506294-9e8f-4aee-9870-1961472ca651-kube-api-access-sltql\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:12.551880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.551876 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70506294-9e8f-4aee-9870-1961472ca651-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:12.552133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:12.551891 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70506294-9e8f-4aee-9870-1961472ca651-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:13.131327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:13.131288 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" event={"ID":"70506294-9e8f-4aee-9870-1961472ca651","Type":"ContainerDied","Data":"2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa"} Oct 30 22:14:13.131514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:13.131354 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e397d610c1493319656a9d035db36c31208e28c2bbd06f1b0dcaa3700f909aa" Oct 30 22:14:13.131514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:13.131256 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7" Oct 30 22:14:16.931326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:16.931288 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:16.935891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:16.935854 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:16.940058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:16.940031 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:14:17.089016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:17.088987 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8"] Oct 30 22:14:17.095148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:17.095121 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5926cbd4-nrxq8"] Oct 30 22:14:18.101826 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:18.101790 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:14:18.199572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:18.199535 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed" path="/var/lib/kubelet/pods/824fcfe5-9e3b-4fbf-80ba-5db3f5c502ed/volumes" Oct 30 22:14:20.614038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:20.614001 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:20.618676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:20.618648 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:20.628148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:20.628120 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:14:21.941292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:21.933853 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" podUID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" containerName="payload-processing" containerID="cri-o://75a3b68aa740cebe30c1638b6d5788aed0c1d7cfa02c5f014cdd8eab08998523" gracePeriod=25 Oct 30 22:14:21.947323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:21.943427 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" podUID="bb855833-6341-4475-bebe-9add129125bb" containerName="payload-processing" containerID="cri-o://80026d2cbd65420aea3a7b2e29a3981accef899ff145bc4b64fd3472ba122575" gracePeriod=25 Oct 30 22:14:22.290053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.289719 5036 generic.go:356] "Generic (PLEG): container finished" podID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" containerID="75a3b68aa740cebe30c1638b6d5788aed0c1d7cfa02c5f014cdd8eab08998523" exitCode=0 Oct 30 22:14:22.290053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.289959 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" event={"ID":"8dbcd5aa-b2db-489e-9110-ee00c86b8e57","Type":"ContainerDied","Data":"75a3b68aa740cebe30c1638b6d5788aed0c1d7cfa02c5f014cdd8eab08998523"} Oct 30 22:14:22.294015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.293731 5036 generic.go:356] "Generic (PLEG): container finished" podID="bb855833-6341-4475-bebe-9add129125bb" containerID="80026d2cbd65420aea3a7b2e29a3981accef899ff145bc4b64fd3472ba122575" exitCode=0 Oct 30 22:14:22.294015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.293962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" event={"ID":"bb855833-6341-4475-bebe-9add129125bb","Type":"ContainerDied","Data":"80026d2cbd65420aea3a7b2e29a3981accef899ff145bc4b64fd3472ba122575"} Oct 30 22:14:22.455144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.453328 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:14:22.458940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.458905 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.643377 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\") pod \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.643544 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\") pod \"bb855833-6341-4475-bebe-9add129125bb\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.643627 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\") pod \"bb855833-6341-4475-bebe-9add129125bb\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.643684 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\") pod \"bb855833-6341-4475-bebe-9add129125bb\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.649737 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\") pod \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.649890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\") pod \"bb855833-6341-4475-bebe-9add129125bb\" (UID: \"bb855833-6341-4475-bebe-9add129125bb\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.649950 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\") pod \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.650019 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\") pod \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\" (UID: \"8dbcd5aa-b2db-489e-9110-ee00c86b8e57\") " Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.651511 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume" pod "8dbcd5aa-b2db-489e-9110-ee00c86b8e57" (UID: "8dbcd5aa-b2db-489e-9110-ee00c86b8e57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.651919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume" pod "bb855833-6341-4475-bebe-9add129125bb" (UID: "bb855833-6341-4475-bebe-9add129125bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.652743 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume" pod "bb855833-6341-4475-bebe-9add129125bb" (UID: "bb855833-6341-4475-bebe-9add129125bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.654084 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.654109 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.654123 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb855833-6341-4475-bebe-9add129125bb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.656894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume" pod "8dbcd5aa-b2db-489e-9110-ee00c86b8e57" (UID: "8dbcd5aa-b2db-489e-9110-ee00c86b8e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.665500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96" pod "bb855833-6341-4475-bebe-9add129125bb" (UID: "bb855833-6341-4475-bebe-9add129125bb"). InnerVolumeSpecName "kube-api-access-gbd96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.665651 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials" pod "8dbcd5aa-b2db-489e-9110-ee00c86b8e57" (UID: "8dbcd5aa-b2db-489e-9110-ee00c86b8e57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:22.666346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.665745 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials" pod "bb855833-6341-4475-bebe-9add129125bb" (UID: "bb855833-6341-4475-bebe-9add129125bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:22.670393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.670347 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk" pod "8dbcd5aa-b2db-489e-9110-ee00c86b8e57" (UID: "8dbcd5aa-b2db-489e-9110-ee00c86b8e57"). InnerVolumeSpecName "kube-api-access-zqpsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:22.755765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.755672 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.755765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.755703 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqpsk\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-kube-api-access-zqpsk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.755765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.755715 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbd96\" (UniqueName: \"kubernetes.io/projected/bb855833-6341-4475-bebe-9add129125bb-kube-api-access-gbd96\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.755765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.755724 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:22.755765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:22.755732 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dbcd5aa-b2db-489e-9110-ee00c86b8e57-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:23.306418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.306388 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" Oct 30 22:14:23.306882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.306439 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq" event={"ID":"8dbcd5aa-b2db-489e-9110-ee00c86b8e57","Type":"ContainerDied","Data":"b26bdcfb6033e018085af79c5489d84933cbe5f948291f551aab1ee709097afe"} Oct 30 22:14:23.306882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.306513 5036 scope.go:122] "RemoveContainer" containerID="75a3b68aa740cebe30c1638b6d5788aed0c1d7cfa02c5f014cdd8eab08998523" Oct 30 22:14:23.308439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.308286 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" Oct 30 22:14:23.308439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.308319 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x" event={"ID":"bb855833-6341-4475-bebe-9add129125bb","Type":"ContainerDied","Data":"8d0ea1ebb3e0b86b27ebe36d4db302c8db0489b2abaddb3d8c398ac52d4b1e7e"} Oct 30 22:14:23.349892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.349865 5036 scope.go:122] "RemoveContainer" containerID="80026d2cbd65420aea3a7b2e29a3981accef899ff145bc4b64fd3472ba122575" Oct 30 22:14:23.375321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.375290 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:23.383208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.383180 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-6484bdzbq"] Oct 30 22:14:23.390192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.390173 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:14:23.403864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:23.403841 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-7478w5v8x"] Oct 30 22:14:24.403383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:24.399222 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" path="/var/lib/kubelet/pods/8dbcd5aa-b2db-489e-9110-ee00c86b8e57/volumes" Oct 30 22:14:24.403383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:24.400064 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb855833-6341-4475-bebe-9add129125bb" path="/var/lib/kubelet/pods/bb855833-6341-4475-bebe-9add129125bb/volumes" Oct 30 22:14:24.627406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:24.620525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:25.624293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:25.622332 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" podUID="4b113ad2-e384-40f2-98c6-c09539b4b82f" containerName="payload-processing" containerID="cri-o://2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f" gracePeriod=25 Oct 30 22:14:25.632297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:25.631329 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" podUID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" containerName="payload-processing" containerID="cri-o://3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e" gracePeriod=25 Oct 30 22:14:26.101902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.101540 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:26.105859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.105387 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:14:26.190346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.189918 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\") pod \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " Oct 30 22:14:26.190346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.189986 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\") pod \"4b113ad2-e384-40f2-98c6-c09539b4b82f\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " Oct 30 22:14:26.190346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\") pod \"4b113ad2-e384-40f2-98c6-c09539b4b82f\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " Oct 30 22:14:26.190673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190352 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\") pod \"4b113ad2-e384-40f2-98c6-c09539b4b82f\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " Oct 30 22:14:26.190673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\") pod \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " Oct 30 22:14:26.190673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190546 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\") pod \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " Oct 30 22:14:26.190673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\") pod \"4b113ad2-e384-40f2-98c6-c09539b4b82f\" (UID: \"4b113ad2-e384-40f2-98c6-c09539b4b82f\") " Oct 30 22:14:26.190890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.190736 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\") pod \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\" (UID: \"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1\") " Oct 30 22:14:26.191858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.191626 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume" pod "4b113ad2-e384-40f2-98c6-c09539b4b82f" (UID: "4b113ad2-e384-40f2-98c6-c09539b4b82f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.191858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.191762 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.192167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.192140 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume" pod "4b113ad2-e384-40f2-98c6-c09539b4b82f" (UID: "4b113ad2-e384-40f2-98c6-c09539b4b82f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.192627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.192600 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume" pod "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" (UID: "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.195151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.195037 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume" pod "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" (UID: "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.201715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.201202 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9" pod "4b113ad2-e384-40f2-98c6-c09539b4b82f" (UID: "4b113ad2-e384-40f2-98c6-c09539b4b82f"). InnerVolumeSpecName "kube-api-access-vcwc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.201715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.201686 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc" pod "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" (UID: "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1"). InnerVolumeSpecName "kube-api-access-9rtlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.201849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.201752 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials" pod "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" (UID: "1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.240291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.209206 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials" pod "4b113ad2-e384-40f2-98c6-c09539b4b82f" (UID: "4b113ad2-e384-40f2-98c6-c09539b4b82f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.293027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.292996 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293025 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwc9\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-kube-api-access-vcwc9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293034 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b113ad2-e384-40f2-98c6-c09539b4b82f-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293044 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtlc\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-kube-api-access-9rtlc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293053 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293061 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b113ad2-e384-40f2-98c6-c09539b4b82f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.293301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.293070 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.357225 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:26.357184 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fd4a0_9c5e_4367_828c_cef3ac3c48e1.slice/crio-39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc\": RecentStats: unable to find data in memory cache]" Oct 30 22:14:26.554345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.553786 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" event={"ID":"4b113ad2-e384-40f2-98c6-c09539b4b82f","Type":"ContainerDied","Data":"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f"} Oct 30 22:14:26.554345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.553870 5036 scope.go:122] "RemoveContainer" containerID="2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f" Oct 30 22:14:26.554345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.554060 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" Oct 30 22:14:26.556587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.553676 5036 generic.go:356] "Generic (PLEG): container finished" podID="4b113ad2-e384-40f2-98c6-c09539b4b82f" containerID="2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f" exitCode=0 Oct 30 22:14:26.557158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.557128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4" event={"ID":"4b113ad2-e384-40f2-98c6-c09539b4b82f","Type":"ContainerDied","Data":"224994271d931f94ad31b2f40ab17a75073a6c1487e054903674acc57e75a74c"} Oct 30 22:14:26.565603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.565559 5036 generic.go:356] "Generic (PLEG): container finished" podID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" containerID="3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e" exitCode=0 Oct 30 22:14:26.565817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.565797 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" event={"ID":"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1","Type":"ContainerDied","Data":"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e"} Oct 30 22:14:26.566011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.565974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" event={"ID":"1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1","Type":"ContainerDied","Data":"39319723827aa0b2e159a61bbe2c57048ba1ca1df99f9bd32d6865d081a8a4dc"} Oct 30 22:14:26.566295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.566254 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw" Oct 30 22:14:26.582092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.582073 5036 scope.go:122] "RemoveContainer" containerID="2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f" Oct 30 22:14:26.582388 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:26.582365 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f\": container with ID starting with 2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f not found: ID does not exist" containerID="2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f" Oct 30 22:14:26.582484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.582398 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f"} err="failed to get container status \"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f\": rpc error: code = NotFound desc = could not find container \"2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f\": container with ID starting with 2cdbf602b41a8d013227b8caf9212341f31b550cf8ed1b6b1c7c0b779b8c721f not found: ID does not exist" Oct 30 22:14:26.582484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.582422 5036 scope.go:122] "RemoveContainer" containerID="3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e" Oct 30 22:14:26.593683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.593662 5036 scope.go:122] "RemoveContainer" containerID="3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e" Oct 30 22:14:26.593972 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:26.593949 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e\": container with ID starting with 3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e not found: ID does not exist" containerID="3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e" Oct 30 22:14:26.594071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.593976 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e"} err="failed to get container status \"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e\": rpc error: code = NotFound desc = could not find container \"3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e\": container with ID starting with 3851088f635c495466157ef024591d305dbb211e102061f5b8854ae553f8ee7e not found: ID does not exist" Oct 30 22:14:26.619101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.619020 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:14:26.641292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.632860 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-85cfv59sw"] Oct 30 22:14:26.658522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.658487 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:26.664326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:26.664299 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a1ea1e/payload-processing-external-model-e2e-extmod-b-a1ea1e-54cdfrtd4"] Oct 30 22:14:27.075398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.075328 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk"] Oct 30 22:14:27.112301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.108338 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99e4748-tzgmk"] Oct 30 22:14:27.561603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.561571 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:14:27.562459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562442 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bb855833-6341-4475-bebe-9add129125bb" containerName="payload-processing" Oct 30 22:14:27.562514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562460 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb855833-6341-4475-bebe-9add129125bb" containerName="payload-processing" Oct 30 22:14:27.562514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562480 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" containerName="payload-processing" Oct 30 22:14:27.562514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562485 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" containerName="payload-processing" Oct 30 22:14:27.562514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562500 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4b113ad2-e384-40f2-98c6-c09539b4b82f" containerName="payload-processing" Oct 30 22:14:27.562514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562505 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b113ad2-e384-40f2-98c6-c09539b4b82f" containerName="payload-processing" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562525 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70506294-9e8f-4aee-9870-1961472ca651" containerName="revoke-keys" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562530 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70506294-9e8f-4aee-9870-1961472ca651" containerName="revoke-keys" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562541 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a210acb-cbd9-4134-be02-a1ce8bc48152" containerName="revoke-keys" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562546 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a210acb-cbd9-4134-be02-a1ce8bc48152" containerName="revoke-keys" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562561 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" containerName="payload-processing" Oct 30 22:14:27.562692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562578 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" containerName="payload-processing" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562698 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70506294-9e8f-4aee-9870-1961472ca651" containerName="revoke-keys" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562708 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a210acb-cbd9-4134-be02-a1ce8bc48152" containerName="revoke-keys" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562715 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbcd5aa-b2db-489e-9110-ee00c86b8e57" containerName="payload-processing" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562724 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb855833-6341-4475-bebe-9add129125bb" containerName="payload-processing" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562734 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" containerName="payload-processing" Oct 30 22:14:27.562884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.562740 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b113ad2-e384-40f2-98c6-c09539b4b82f" containerName="payload-processing" Oct 30 22:14:27.568400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.568377 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.573537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.573514 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-813840a060cc\"" Oct 30 22:14:27.583470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.574402 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\"" Oct 30 22:14:27.640107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.640061 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.669554 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.669725 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2bz\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.669848 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.669984 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.678526 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\"" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.678637 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\"" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.678709 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-3aae80dfab12\"" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.678768 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\"" Oct 30 22:14:27.682292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.679078 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-zdlx9\"" Oct 30 22:14:27.771131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.771041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.771360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.771141 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2bz\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.771360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.771205 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.771360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.771281 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.771967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.771885 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.772116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.772027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.773951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.773929 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.783039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.783016 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2bz\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:27.957528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:27.957490 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:28.193941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.193906 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1" path="/var/lib/kubelet/pods/1c5fd4a0-9c5e-4367-828c-cef3ac3c48e1/volumes" Oct 30 22:14:28.194406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.194390 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b113ad2-e384-40f2-98c6-c09539b4b82f" path="/var/lib/kubelet/pods/4b113ad2-e384-40f2-98c6-c09539b4b82f/volumes" Oct 30 22:14:28.194876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.194847 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a60d37-01d1-4a34-9323-3bed2efc638e" path="/var/lib/kubelet/pods/64a60d37-01d1-4a34-9323-3bed2efc638e/volumes" Oct 30 22:14:28.387403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.387379 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:14:28.744324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.735437 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" event={"ID":"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df","Type":"ContainerStarted","Data":"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825"} Oct 30 22:14:28.744324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.735532 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" event={"ID":"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df","Type":"ContainerStarted","Data":"b28168651eb618dd37504028b7329c525e1633e435a386ad546e54d237863fe6"} Oct 30 22:14:28.744324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.738008 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:28.763814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:28.763716 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" podStartSLOduration=1.763698762 podStartE2EDuration="1.763698762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:28.761631114 +0000 UTC m=+2092.711586059" watchObservedRunningTime="2026-10-30 22:14:28.763698762 +0000 UTC m=+2092.713653709" Oct 30 22:14:32.422291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.421447 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:14:32.481289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.476578 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.524812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.524717 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:14:32.577292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.572895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.577292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.572941 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.577292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.573017 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.577292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.573079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx665\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.680356 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.680451 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx665\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.680808 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.680836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.681471 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.686335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.682233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.697292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.686989 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.700826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.700755 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx665\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:32.883534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:32.883494 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:33.186417 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:33.186373 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6681db39_45c7_45ea_85ce_d4c6f64daa2d.slice/crio-80941c84f796c4552f19a2f66f9e10a7d45d03f2e9e387b57218f6600f35ba41 WatchSource:0}: Error finding container 80941c84f796c4552f19a2f66f9e10a7d45d03f2e9e387b57218f6600f35ba41: Status 404 returned error can't find the container with id 80941c84f796c4552f19a2f66f9e10a7d45d03f2e9e387b57218f6600f35ba41 Oct 30 22:14:33.186738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:33.186714 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:14:33.795507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:33.795462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" event={"ID":"6681db39-45c7-45ea-85ce-d4c6f64daa2d","Type":"ContainerStarted","Data":"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613"} Oct 30 22:14:33.796070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:33.795553 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" event={"ID":"6681db39-45c7-45ea-85ce-d4c6f64daa2d","Type":"ContainerStarted","Data":"80941c84f796c4552f19a2f66f9e10a7d45d03f2e9e387b57218f6600f35ba41"} Oct 30 22:14:33.796070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:33.795576 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:34.850206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.850170 5036 generic.go:356] "Generic (PLEG): container finished" podID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerID="5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4" exitCode=0 Oct 30 22:14:34.854380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.853457 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerDied","Data":"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4"} Oct 30 22:14:34.855034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.855014 5036 scope.go:122] "RemoveContainer" containerID="5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4" Oct 30 22:14:34.860431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.860385 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:34.861515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.861485 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:34.861698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.861677 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:34.897755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:34.897696 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" podStartSLOduration=2.897674619 podStartE2EDuration="2.897674619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:33.815579944 +0000 UTC m=+2097.765534892" watchObservedRunningTime="2026-10-30 22:14:34.897674619 +0000 UTC m=+2098.847629566" Oct 30 22:14:35.918350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:35.915837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerStarted","Data":"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460"} Oct 30 22:14:36.384672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.384645 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:36.389759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.389736 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.397222 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-56549985aa0b\"" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.397414 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9d868\"" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.397525 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"routing-overlay\"" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.397798 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"kube-root-ca.crt\"" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.397845 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"openshift-service-ca.crt\"" Oct 30 22:14:36.399339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.398470 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-e2e--2051f056-dockercfg-f82w9\"" Oct 30 22:14:36.407876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.407852 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:36.854107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.853918 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:14:36.854674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.854641 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:36.854860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.854835 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:36.873970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.864333 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.873970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.864480 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.873970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.864641 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.873970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.864785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6l4\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.986833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.986799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.987346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.986962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.987346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.987080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6l4\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.987346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.987323 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.987764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.987730 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.988017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.987996 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:36.990120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:36.990098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:37.002667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.002646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6l4\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:37.007340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.007321 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:14:37.007636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.007618 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:14:37.263289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.262708 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:37.598028 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:37.597992 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d157f85_2e6e_4416_a2d4_d8a87a55923f.slice/crio-17caaa55381672dbad4a9748855423f1b844617cb7440d5be65971c4166630fe WatchSource:0}: Error finding container 17caaa55381672dbad4a9748855423f1b844617cb7440d5be65971c4166630fe: Status 404 returned error can't find the container with id 17caaa55381672dbad4a9748855423f1b844617cb7440d5be65971c4166630fe Oct 30 22:14:37.598716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.598632 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:37.830163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.829740 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:37.830163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.829795 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:37.969185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.969147 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" event={"ID":"1d157f85-2e6e-4416-a2d4-d8a87a55923f","Type":"ContainerStarted","Data":"6fcf87ff4beaea4b23f491d555033cf9afaa9bdcd2623aaef95abd0a15169b75"} Oct 30 22:14:37.969435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.969213 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" event={"ID":"1d157f85-2e6e-4416-a2d4-d8a87a55923f","Type":"ContainerStarted","Data":"17caaa55381672dbad4a9748855423f1b844617cb7440d5be65971c4166630fe"} Oct 30 22:14:37.969435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:37.969245 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:38.050439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:38.049285 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" podStartSLOduration=2.049249592 podStartE2EDuration="2.049249592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:38.039538428 +0000 UTC m=+2101.989493375" watchObservedRunningTime="2026-10-30 22:14:38.049249592 +0000 UTC m=+2101.999204539" Oct 30 22:14:38.838297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:38.836643 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:38.838297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:38.836729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:39.118342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.109368 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:39.181969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.179218 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:14:39.263306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.208770 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:39.395357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.381416 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:14:39.396370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.395659 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:14:39.427690 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.427649 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:39.427952 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.427932 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:39.428197 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.428175 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:39.92815036 +0000 UTC m=+2103.878105298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:39.602857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.602494 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:39.794294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.794181 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:14:39.865018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.864947 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:39.865018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:39.865010 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:39.996464 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.996437 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:39.996464 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.996458 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:39.996701 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:39.996504 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:40.99648839 +0000 UTC m=+2104.946443315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:40.577758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.577714 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:14:40.588415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.588387 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:14:40.588415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.588434 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:14:40.588620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.588521 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.593988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.593966 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:14:40.594123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.594043 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.599052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.599024 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"e2e-cmp-legacy-5dd151-provider-cred\"" Oct 30 22:14:40.599196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.599110 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"e2e-cfl-6cd340-provider-cred\"" Oct 30 22:14:40.599749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.599384 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-e2e--6ed667c1-dockercfg-zrb67\"" Oct 30 22:14:40.603852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.603816 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"kube-root-ca.crt\"" Oct 30 22:14:40.604070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.604044 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bacea0\"" Oct 30 22:14:40.604174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.604080 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"routing-overlay\"" Oct 30 22:14:40.604174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.604131 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"openshift-service-ca.crt\"" Oct 30 22:14:40.735520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.735476 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.741607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741550 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.741797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mln\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.741875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbsc\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.741875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.741875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.742064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741906 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.742064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.741930 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.847847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.847744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbsc\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.858763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.858729 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.860012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.859985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.860780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.860758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.860943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.860926 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.861308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.861291 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.861648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.861629 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.862319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.859901 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.872542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.872513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mln\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.874615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.874527 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:40.874793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.874639 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:40.876385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.874605 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.878889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.878842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.879234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.879214 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.885302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.880357 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.885432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.885350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbsc\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.896321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.892177 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.896321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.893839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mln\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:40.909371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.909350 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:40.942698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:40.942660 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:41.077425 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:41.077393 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:41.077425 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:41.077425 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:41.077615 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:41.077489 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:43.077468569 +0000 UTC m=+2107.027423493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:41.253277 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:41.253208 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585c658b_ba8c_4fc6_a1c8_16f55d4519e1.slice/crio-efe635a1024965d2d6dc8ede3af12f53451ed3267552765c4090c49d8aeb49e2 WatchSource:0}: Error finding container efe635a1024965d2d6dc8ede3af12f53451ed3267552765c4090c49d8aeb49e2: Status 404 returned error can't find the container with id efe635a1024965d2d6dc8ede3af12f53451ed3267552765c4090c49d8aeb49e2 Oct 30 22:14:41.255735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.255441 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:14:41.281840 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:41.281813 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d78f7b_a040_45ae_9e84_e40ac20bd801.slice/crio-787f0be79aba3e103350cb1030f23c9968236bfd9e6b4786caa7706661528c45 WatchSource:0}: Error finding container 787f0be79aba3e103350cb1030f23c9968236bfd9e6b4786caa7706661528c45: Status 404 returned error can't find the container with id 787f0be79aba3e103350cb1030f23c9968236bfd9e6b4786caa7706661528c45 Oct 30 22:14:41.283506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.283431 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:14:41.600907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.600807 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" event={"ID":"585c658b-ba8c-4fc6-a1c8-16f55d4519e1","Type":"ContainerStarted","Data":"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f"} Oct 30 22:14:41.600907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.600865 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" event={"ID":"585c658b-ba8c-4fc6-a1c8-16f55d4519e1","Type":"ContainerStarted","Data":"efe635a1024965d2d6dc8ede3af12f53451ed3267552765c4090c49d8aeb49e2"} Oct 30 22:14:41.600907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.600894 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:41.602931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.602906 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" event={"ID":"30d78f7b-a040-45ae-9e84-e40ac20bd801","Type":"ContainerStarted","Data":"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a"} Oct 30 22:14:41.603063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.602956 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" event={"ID":"30d78f7b-a040-45ae-9e84-e40ac20bd801","Type":"ContainerStarted","Data":"787f0be79aba3e103350cb1030f23c9968236bfd9e6b4786caa7706661528c45"} Oct 30 22:14:41.603063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.603041 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:41.672317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.669646 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" podStartSLOduration=1.6696199900000002 podStartE2EDuration="1.66961999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:41.634766278 +0000 UTC m=+2105.584721226" watchObservedRunningTime="2026-10-30 22:14:41.66961999 +0000 UTC m=+2105.619574937" Oct 30 22:14:41.716294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.715421 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" podStartSLOduration=1.7153998719999999 podStartE2EDuration="1.715399872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:41.709603526 +0000 UTC m=+2105.659558513" watchObservedRunningTime="2026-10-30 22:14:41.715399872 +0000 UTC m=+2105.665354819" Oct 30 22:14:41.848373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.845956 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:41.848373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:41.846017 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:42.836658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:42.836616 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 30 22:14:42.837252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:42.837217 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 30 22:14:43.162289 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:43.156221 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:43.162289 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:43.156278 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:43.162289 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:43.156355 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:47.156335395 +0000 UTC m=+2111.106290335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:43.704112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:43.704077 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:14:43.836146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:43.836112 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:14:44.680232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:44.680202 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:14:44.872472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:44.872440 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:14:44.951324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:44.951219 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:14:46.032001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:46.031911 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k"] Oct 30 22:14:46.039409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:46.039377 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a606d0d-98z7k"] Oct 30 22:14:46.192888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:46.192856 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58db2825-c2ac-4c18-98f2-815e691bf2e5" path="/var/lib/kubelet/pods/58db2825-c2ac-4c18-98f2-815e691bf2e5/volumes" Oct 30 22:14:46.376367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:46.375114 5036 scope.go:122] "RemoveContainer" containerID="df03d8ab0d165277f94ec9b903450776bad07525e618d0edd278587122787425" Oct 30 22:14:47.257512 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:47.257476 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:47.257512 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:47.257508 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:47.257972 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:47.257578 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:14:55.257564221 +0000 UTC m=+2119.207519146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:48.103831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:48.103793 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:14:49.250186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:49.250155 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:49.952628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:49.952585 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" podUID="45a463b7-e764-4f43-be0c-6d1903f6f522" containerName="payload-processing" containerID="cri-o://158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3" gracePeriod=25 Oct 30 22:14:50.259757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.259732 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:14:50.328911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.328871 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\") pod \"45a463b7-e764-4f43-be0c-6d1903f6f522\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " Oct 30 22:14:50.329107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.329082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\") pod \"45a463b7-e764-4f43-be0c-6d1903f6f522\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " Oct 30 22:14:50.329182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.329168 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\") pod \"45a463b7-e764-4f43-be0c-6d1903f6f522\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " Oct 30 22:14:50.329243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.329225 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume" pod "45a463b7-e764-4f43-be0c-6d1903f6f522" (UID: "45a463b7-e764-4f43-be0c-6d1903f6f522"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:50.329379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.329345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\") pod \"45a463b7-e764-4f43-be0c-6d1903f6f522\" (UID: \"45a463b7-e764-4f43-be0c-6d1903f6f522\") " Oct 30 22:14:50.329783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.329759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume" pod "45a463b7-e764-4f43-be0c-6d1903f6f522" (UID: "45a463b7-e764-4f43-be0c-6d1903f6f522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:50.330887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.330677 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:50.330887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.330701 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a463b7-e764-4f43-be0c-6d1903f6f522-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:50.333164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.332102 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk" pod "45a463b7-e764-4f43-be0c-6d1903f6f522" (UID: "45a463b7-e764-4f43-be0c-6d1903f6f522"). InnerVolumeSpecName "kube-api-access-vntzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:50.333164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.332442 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials" pod "45a463b7-e764-4f43-be0c-6d1903f6f522" (UID: "45a463b7-e764-4f43-be0c-6d1903f6f522"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:50.432923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.432802 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntzk\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-kube-api-access-vntzk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:50.433238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.433215 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a463b7-e764-4f43-be0c-6d1903f6f522-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:50.752170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.752134 5036 generic.go:356] "Generic (PLEG): container finished" podID="45a463b7-e764-4f43-be0c-6d1903f6f522" containerID="158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3" exitCode=0 Oct 30 22:14:50.752433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.752173 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" event={"ID":"45a463b7-e764-4f43-be0c-6d1903f6f522","Type":"ContainerDied","Data":"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3"} Oct 30 22:14:50.752433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.752204 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" Oct 30 22:14:50.752433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.752243 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw" event={"ID":"45a463b7-e764-4f43-be0c-6d1903f6f522","Type":"ContainerDied","Data":"ee40d5c9c86e818cfe02faa84f844454366f897da4bed218b12ee7c4776590f8"} Oct 30 22:14:50.752433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.752287 5036 scope.go:122] "RemoveContainer" containerID="158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3" Oct 30 22:14:50.767581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.767549 5036 scope.go:122] "RemoveContainer" containerID="158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3" Oct 30 22:14:50.767844 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:50.767821 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3\": container with ID starting with 158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3 not found: ID does not exist" containerID="158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3" Oct 30 22:14:50.767940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.767854 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3"} err="failed to get container status \"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3\": rpc error: code = NotFound desc = could not find container \"158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3\": container with ID starting with 158f5d1fbe3bbab7ecbf231fbbfd954de3c7b67f1185677236d69cf33b4807b3 not found: ID does not exist" Oct 30 22:14:50.786559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.786539 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:14:50.792237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:50.792218 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-64d6fdsxw"] Oct 30 22:14:51.155461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.155426 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:14:51.156849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.156827 5036 cpu_manager.go:414] "containerMap: removing container" podUID="45a463b7-e764-4f43-be0c-6d1903f6f522" containerName="payload-processing" Oct 30 22:14:51.156919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.156851 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a463b7-e764-4f43-be0c-6d1903f6f522" containerName="payload-processing" Oct 30 22:14:51.157136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.157123 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a463b7-e764-4f43-be0c-6d1903f6f522" containerName="payload-processing" Oct 30 22:14:51.163211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.163191 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.171221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.171197 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-26a128-openshift-default-dockercfg-7xjnf\"" Oct 30 22:14:51.180238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.180217 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:14:51.245915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.245874 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.245934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clqj\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246187 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246291 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246371 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246420 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246448 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.246544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.246530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.347997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.347949 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348251 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348311 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clqj\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348367 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348648 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348661 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348726 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.348946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.348922 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.349242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.349223 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.349338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.349239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.351447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.351418 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.351542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.351482 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.360582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.360557 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.361081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.361061 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clqj\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.482530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.482444 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:51.888068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.888036 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:14:51.888498 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:51.888478 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b289c4_9657_4f1e_83e9_2b8a91c40af0.slice/crio-e78955b27bd0903249574aa973693c260e4a8acb0a770c90f2fc00cb39643a65 WatchSource:0}: Error finding container e78955b27bd0903249574aa973693c260e4a8acb0a770c90f2fc00cb39643a65: Status 404 returned error can't find the container with id e78955b27bd0903249574aa973693c260e4a8acb0a770c90f2fc00cb39643a65 Oct 30 22:14:51.891767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.891727 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:51.891868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.891814 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:51.891868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:51.891858 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:52.192445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.192360 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a463b7-e764-4f43-be0c-6d1903f6f522" path="/var/lib/kubelet/pods/45a463b7-e764-4f43-be0c-6d1903f6f522/volumes" Oct 30 22:14:52.648174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.648145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:14:52.648625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.648233 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:14:52.730617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.730581 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:52.780026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.779982 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" event={"ID":"31b289c4-9657-4f1e-83e9-2b8a91c40af0","Type":"ContainerStarted","Data":"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384"} Oct 30 22:14:52.780232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.780056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" event={"ID":"31b289c4-9657-4f1e-83e9-2b8a91c40af0","Type":"ContainerStarted","Data":"e78955b27bd0903249574aa973693c260e4a8acb0a770c90f2fc00cb39643a65"} Oct 30 22:14:52.806152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:52.806088 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podStartSLOduration=1.8060703340000002 podStartE2EDuration="1.806070334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:52.802915289 +0000 UTC m=+2116.752870248" watchObservedRunningTime="2026-10-30 22:14:52.806070334 +0000 UTC m=+2116.756025281" Oct 30 22:14:53.485313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:53.483737 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:53.485313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:53.485006 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:53.485313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:53.485059 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:53.973286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:53.971900 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:14:54.021290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.020688 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.165337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.164978 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.165337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.165146 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.165337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.165327 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxldl\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.165682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.165535 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.184299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.183331 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:14:54.267258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.267171 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.267448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.267366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxldl\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.267890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.267855 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.268505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.268478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.268505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.268491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.269094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.269074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.270383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.270187 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.279350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.279327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxldl\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:54.482954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.482917 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:54.483226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.482975 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:54.537370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:54.535567 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:55.268480 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:14:55.268444 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f86e877_9b24_44fa_8607_1f6b75fc7f46.slice/crio-dcdfe5dae0859a6da26859087eee511b460823b6374cbc3edbba2735784d66d0 WatchSource:0}: Error finding container dcdfe5dae0859a6da26859087eee511b460823b6374cbc3edbba2735784d66d0: Status 404 returned error can't find the container with id dcdfe5dae0859a6da26859087eee511b460823b6374cbc3edbba2735784d66d0 Oct 30 22:14:55.268818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:55.268534 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:14:55.350316 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:55.349299 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-56549985aa0b: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:55.350316 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:55.349326 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc: secret "agc-cred-56549985aa0b" not found Oct 30 22:14:55.350316 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:14:55.349383 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials podName:1d157f85-2e6e-4416-a2d4-d8a87a55923f nodeName:}" failed. No retries permitted until 2026-10-30 22:15:11.349362675 +0000 UTC m=+2135.299317794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f") : secret "agc-cred-56549985aa0b" not found Oct 30 22:14:55.483734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:55.483703 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:55.483904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:55.483758 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:56.055149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.055094 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" event={"ID":"7f86e877-9b24-44fa-8607-1f6b75fc7f46","Type":"ContainerStarted","Data":"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a"} Oct 30 22:14:56.055513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.055180 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" event={"ID":"7f86e877-9b24-44fa-8607-1f6b75fc7f46","Type":"ContainerStarted","Data":"dcdfe5dae0859a6da26859087eee511b460823b6374cbc3edbba2735784d66d0"} Oct 30 22:14:56.056398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.056367 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:14:56.079253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.079207 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" podStartSLOduration=3.079192749 podStartE2EDuration="3.079192749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:56.074426607 +0000 UTC m=+2120.024381556" watchObservedRunningTime="2026-10-30 22:14:56.079192749 +0000 UTC m=+2120.029147695" Oct 30 22:14:56.483564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.483532 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:56.483986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.483581 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:56.957195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.957159 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm"] Oct 30 22:14:56.964125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.964098 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:56.965597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.965569 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm"] Oct 30 22:14:56.970084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.970061 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:56.970493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.970469 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:56.970724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.970704 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:14:56.970782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:56.970767 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:57.053465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.052059 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm"] Oct 30 22:14:57.060460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.060422 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd21abd-86qpm"] Oct 30 22:14:57.083384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.083357 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.083537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.083402 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.083583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.083568 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8thp\" (UniqueName: \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.194842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.194792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8thp\" (UniqueName: \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.196401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.196373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.197647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.197624 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.198646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.198621 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.201153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.201111 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.220326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.220230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8thp\" (UniqueName: \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\") pod \"maas-api-revoke-sub-6b02ad45-mhpgm\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.282870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.282719 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:14:57.482882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.482799 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:57.482882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.482864 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:57.646195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.646154 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm"] Oct 30 22:14:57.736784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:57.736462 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" podUID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" containerName="payload-processing" containerID="cri-o://6fcf87ff4beaea4b23f491d555033cf9afaa9bdcd2623aaef95abd0a15169b75" gracePeriod=25 Oct 30 22:14:58.172326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.157252 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" containerID="6fcf87ff4beaea4b23f491d555033cf9afaa9bdcd2623aaef95abd0a15169b75" exitCode=0 Oct 30 22:14:58.172326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.157474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" event={"ID":"1d157f85-2e6e-4416-a2d4-d8a87a55923f","Type":"ContainerDied","Data":"6fcf87ff4beaea4b23f491d555033cf9afaa9bdcd2623aaef95abd0a15169b75"} Oct 30 22:14:58.172326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.165079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" event={"ID":"5cd1153d-da27-45b4-8da6-2f9a20f68348","Type":"ContainerStarted","Data":"6e9e095243c7b2a8bc8a6f3ad0518c8fa703c5fecbcec36c3812f067586d2a03"} Oct 30 22:14:58.197858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.197823 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f601b29d-3946-4486-af58-b3a9db116537" path="/var/lib/kubelet/pods/f601b29d-3946-4486-af58-b3a9db116537/volumes" Oct 30 22:14:58.300313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.299066 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:58.371892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.371811 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\") pod \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " Oct 30 22:14:58.372077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.371897 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\") pod \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " Oct 30 22:14:58.372077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.371984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\") pod \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " Oct 30 22:14:58.372224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.372099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\") pod \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\" (UID: \"1d157f85-2e6e-4416-a2d4-d8a87a55923f\") " Oct 30 22:14:58.372407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.372381 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume" pod "1d157f85-2e6e-4416-a2d4-d8a87a55923f" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:58.372500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.372479 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume" pod "1d157f85-2e6e-4416-a2d4-d8a87a55923f" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:58.372789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.372770 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.372789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.372788 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d157f85-2e6e-4416-a2d4-d8a87a55923f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.374602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.374571 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4" pod "1d157f85-2e6e-4416-a2d4-d8a87a55923f" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f"). InnerVolumeSpecName "kube-api-access-8g6l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.374886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.374859 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials" pod "1d157f85-2e6e-4416-a2d4-d8a87a55923f" (UID: "1d157f85-2e6e-4416-a2d4-d8a87a55923f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.475063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.475022 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6l4\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-kube-api-access-8g6l4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.475063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.475065 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d157f85-2e6e-4416-a2d4-d8a87a55923f-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.483804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.483766 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 22:14:58.483933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:58.483821 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 22:14:59.176707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.176673 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" Oct 30 22:14:59.177330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.176696 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc" event={"ID":"1d157f85-2e6e-4416-a2d4-d8a87a55923f","Type":"ContainerDied","Data":"17caaa55381672dbad4a9748855423f1b844617cb7440d5be65971c4166630fe"} Oct 30 22:14:59.177448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.177362 5036 scope.go:122] "RemoveContainer" containerID="6fcf87ff4beaea4b23f491d555033cf9afaa9bdcd2623aaef95abd0a15169b75" Oct 30 22:14:59.178800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.178760 5036 generic.go:356] "Generic (PLEG): container finished" podID="5cd1153d-da27-45b4-8da6-2f9a20f68348" containerID="a8c56f641e34972df11bf31e5219c000d40d72088c85c62986e5d91b3e184388" exitCode=0 Oct 30 22:14:59.179464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.179426 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" event={"ID":"5cd1153d-da27-45b4-8da6-2f9a20f68348","Type":"ContainerDied","Data":"a8c56f641e34972df11bf31e5219c000d40d72088c85c62986e5d91b3e184388"} Oct 30 22:14:59.221429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.221401 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:59.230941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.230908 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9cgwxgc"] Oct 30 22:14:59.565577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.564829 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:59.565770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.565699 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:14:59.566881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:14:59.566858 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:15:00.192541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.192338 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" path="/var/lib/kubelet/pods/1d157f85-2e6e-4416-a2d4-d8a87a55923f/volumes" Oct 30 22:15:00.193931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.193907 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:00.195149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.195125 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" containerName="payload-processing" Oct 30 22:15:00.195149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.195148 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" containerName="payload-processing" Oct 30 22:15:00.195357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.195335 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d157f85-2e6e-4416-a2d4-d8a87a55923f" containerName="payload-processing" Oct 30 22:15:00.201056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.201028 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.201178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.201059 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:00.206295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.206248 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.210371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.209938 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:00.249289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.226499 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.289306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.289219 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:00.297659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.296002 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:00.298993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.298217 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.365759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.365720 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.365939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.365789 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmks\" (UniqueName: \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.365939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.365853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.365939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.365894 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5k8\" (UniqueName: \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.366122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.366017 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.366122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.366040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbrm\" (UniqueName: \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.366242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.366125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.366242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.366156 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znp94\" (UniqueName: \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.373168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.373137 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:00.373409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.373292 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.385985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.384560 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:00.397013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.396982 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:00.412305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.412278 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:00.412636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.412570 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.419525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.417898 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.453903 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.454033 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.469402 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znp94\" (UniqueName: \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475299 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmks\" (UniqueName: \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475396 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.475455 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5k8\" (UniqueName: \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.476737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.476833 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.476869 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbrm\" (UniqueName: \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.476911 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.476994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.477057 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbbn\" (UniqueName: \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.477111 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfl7\" (UniqueName: \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.478057 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.478891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.480634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.479556 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.488474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.484566 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:00.493479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.493453 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.494570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.494375 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:00.522812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.522788 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:00.523167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.523152 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:00.526421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.523094 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.527778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.527713 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5k8\" (UniqueName: \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.528499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.527910 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znp94\" (UniqueName: \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.528499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.528457 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmks\" (UniqueName: \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.529555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.529536 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:00.531935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.531908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbrm\" (UniqueName: \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.534882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.534688 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:00.544711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.544693 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:00.544969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.544900 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.547034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.547017 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:00.564847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.564812 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:00.571851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.571696 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.575480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.575457 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:00.578739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578711 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwhj\" (UniqueName: \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.578885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578799 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.578885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578829 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv884\" (UniqueName: \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.578885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.579059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578901 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.579059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578947 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.579059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.578985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbbn\" (UniqueName: \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.579059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.579016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfl7\" (UniqueName: \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.580087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.580062 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.580608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.580587 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.584526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.584466 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:00.588073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.588051 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:00.592959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.592130 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:00.598499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.598333 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:00.598616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.598446 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.605848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.605826 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.616991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.616957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbbn\" (UniqueName: \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.638495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.638463 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfl7\" (UniqueName: \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.648611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.648583 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:00.659643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.659612 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:00.669995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.669973 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:00.671583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.671558 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:15:00.672531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.672506 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5cd1153d-da27-45b4-8da6-2f9a20f68348" containerName="revoke-keys" Oct 30 22:15:00.673968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.673948 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd1153d-da27-45b4-8da6-2f9a20f68348" containerName="revoke-keys" Oct 30 22:15:00.674691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.674672 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd1153d-da27-45b4-8da6-2f9a20f68348" containerName="revoke-keys" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwhj\" (UniqueName: \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681231 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681303 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxkh\" (UniqueName: \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681394 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv884\" (UniqueName: \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681565 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681641 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4jn\" (UniqueName: \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681744 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prslk\" (UniqueName: \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681817 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sr5\" (UniqueName: \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.681961 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcps5\" (UniqueName: \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.682038 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.682080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.682105 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.684489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.683612 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.685850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.685392 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:00.686040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.686009 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.688564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.688545 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:00.688709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.688589 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.695577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.695397 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:00.710578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.704750 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:00.713922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.713765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwhj\" (UniqueName: \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\") pod \"maas-api-key-cleanup-29889975-lqx56\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.746854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.732785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv884\" (UniqueName: \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.757826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.757583 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:00.770670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.770077 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:00.778515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.778443 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:00.783371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.783117 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\") pod \"5cd1153d-da27-45b4-8da6-2f9a20f68348\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.783670 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\") pod \"5cd1153d-da27-45b4-8da6-2f9a20f68348\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.783746 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\" (UniqueName: \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\") pod \"5cd1153d-da27-45b4-8da6-2f9a20f68348\" (UID: \"5cd1153d-da27-45b4-8da6-2f9a20f68348\") " Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784232 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4jn\" (UniqueName: \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784355 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prslk\" (UniqueName: \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vwv\" (UniqueName: \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sr5\" (UniqueName: \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784617 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcps5\" (UniqueName: \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784739 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784766 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784835 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784894 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxkh\" (UniqueName: \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.784945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.785296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.785103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.791483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.790785 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca" pod "5cd1153d-da27-45b4-8da6-2f9a20f68348" (UID: "5cd1153d-da27-45b4-8da6-2f9a20f68348"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:00.795956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.794619 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.795956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.795245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.795956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.795921 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:00.796812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.795982 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp" pod "5cd1153d-da27-45b4-8da6-2f9a20f68348" (UID: "5cd1153d-da27-45b4-8da6-2f9a20f68348"). InnerVolumeSpecName "kube-api-access-h8thp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:00.796812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.796041 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.796922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.796865 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.797544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.797466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca" pod "5cd1153d-da27-45b4-8da6-2f9a20f68348" (UID: "5cd1153d-da27-45b4-8da6-2f9a20f68348"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:00.805141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.805114 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.811928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.811902 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:00.861223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.860805 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxkh\" (UniqueName: \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:00.861223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.861004 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prslk\" (UniqueName: \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.871297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.867895 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcps5\" (UniqueName: \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\") pod \"maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:00.871297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.867921 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4jn\" (UniqueName: \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.875478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.871947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sr5\" (UniqueName: \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:00.888896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.888863 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:00.890292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.889839 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vwv\" (UniqueName: \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.890292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.889907 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:00.892808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.891792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v6g\" (UniqueName: \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:00.892808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.891950 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.892808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.892360 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:00.892808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.892436 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8thp\" (UniqueName: \"kubernetes.io/projected/5cd1153d-da27-45b4-8da6-2f9a20f68348-kube-api-access-h8thp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:00.892808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.892459 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd1153d-da27-45b4-8da6-2f9a20f68348-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:00.895847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.895643 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:00.912692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.912626 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.913557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.913535 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vwv\" (UniqueName: \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.940146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.939947 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:00.972334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.971986 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:00.995220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.995179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v6g\" (UniqueName: \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:00.995617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.995589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:00.999938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:00.999912 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:01.017913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.017885 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:01.074437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.074393 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v6g\" (UniqueName: \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:01.083187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.082949 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:01.098865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.098664 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:01.109846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.109808 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:01.236295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.234574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerStarted","Data":"8633d0fa86b0460bcc81b26c31ceb191607677e2fe4af78e0a12267b3a125999"} Oct 30 22:15:01.239453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.237604 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" Oct 30 22:15:01.239453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.238091 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm" event={"ID":"5cd1153d-da27-45b4-8da6-2f9a20f68348","Type":"ContainerDied","Data":"6e9e095243c7b2a8bc8a6f3ad0518c8fa703c5fecbcec36c3812f067586d2a03"} Oct 30 22:15:01.239453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.238140 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9e095243c7b2a8bc8a6f3ad0518c8fa703c5fecbcec36c3812f067586d2a03" Oct 30 22:15:01.239453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.238381 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:15:01.264067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.250985 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:01.468164 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.468117 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c761ec4_d3b0_4663_88e7_2de45be4f9f7.slice/crio-7a8ce4a7faf5409c106bf9e6777d5541005fd82b36d92975bc7c66f1b28b98c4 WatchSource:0}: Error finding container 7a8ce4a7faf5409c106bf9e6777d5541005fd82b36d92975bc7c66f1b28b98c4: Status 404 returned error can't find the container with id 7a8ce4a7faf5409c106bf9e6777d5541005fd82b36d92975bc7c66f1b28b98c4 Oct 30 22:15:01.492960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.492860 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:01.654560 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.654526 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf049fb_3442_41b6_abe4_b43daeaf22e2.slice/crio-8b207bac9199e8f36bbfb67cd94b6ec553039640cfce777c5a55f6dde974542a WatchSource:0}: Error finding container 8b207bac9199e8f36bbfb67cd94b6ec553039640cfce777c5a55f6dde974542a: Status 404 returned error can't find the container with id 8b207bac9199e8f36bbfb67cd94b6ec553039640cfce777c5a55f6dde974542a Oct 30 22:15:01.658194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.658152 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:01.704199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.704172 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:01.747612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.747586 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:01.783350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.783310 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:01.831575 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.827035 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bd0e01_0001_426e_b7a3_e8140661136f.slice/crio-6d8abf7f9fecc54f493b52bb038179aa68ecd4eafde98a9fafb9e5afbe895019 WatchSource:0}: Error finding container 6d8abf7f9fecc54f493b52bb038179aa68ecd4eafde98a9fafb9e5afbe895019: Status 404 returned error can't find the container with id 6d8abf7f9fecc54f493b52bb038179aa68ecd4eafde98a9fafb9e5afbe895019 Oct 30 22:15:01.833812 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.833499 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8527b219_c5dc_45ae_b088_0db98e8e1042.slice/crio-abe1e6c826de10c06a98579f00c35cf679fd4eb85585d809181d5403a348192d WatchSource:0}: Error finding container abe1e6c826de10c06a98579f00c35cf679fd4eb85585d809181d5403a348192d: Status 404 returned error can't find the container with id abe1e6c826de10c06a98579f00c35cf679fd4eb85585d809181d5403a348192d Oct 30 22:15:01.835790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.835573 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:01.850232 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.850189 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa20ee_bcfc_460f_bdc2_2a8f34d2176e.slice/crio-281dd6d624e68501dc7f1eee513db2ab555bc1553cbb63f51ae469815575a2d4 WatchSource:0}: Error finding container 281dd6d624e68501dc7f1eee513db2ab555bc1553cbb63f51ae469815575a2d4: Status 404 returned error can't find the container with id 281dd6d624e68501dc7f1eee513db2ab555bc1553cbb63f51ae469815575a2d4 Oct 30 22:15:01.880301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.880251 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:01.901359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.901334 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:01.915921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.915895 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:01.926619 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:01.926581 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13459fce_bfbc_45e0_8e1a_7c6b567c76e9.slice/crio-612680b46f103ddcbd9b9676e383143ebfb7f7d789f7606a846a049cc050f6dc WatchSource:0}: Error finding container 612680b46f103ddcbd9b9676e383143ebfb7f7d789f7606a846a049cc050f6dc: Status 404 returned error can't find the container with id 612680b46f103ddcbd9b9676e383143ebfb7f7d789f7606a846a049cc050f6dc Oct 30 22:15:01.946814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.946793 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:01.956393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.956366 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:01.962508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.962482 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:01.978834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:01.978798 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:02.169005 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:02.168956 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd97a033_6160_4dc2_bde2_0ab637df4548.slice/crio-2447be663b64590b840fb7220572c1c90d1faa9b76032e4d5b3ad8f76f24b6a2 WatchSource:0}: Error finding container 2447be663b64590b840fb7220572c1c90d1faa9b76032e4d5b3ad8f76f24b6a2: Status 404 returned error can't find the container with id 2447be663b64590b840fb7220572c1c90d1faa9b76032e4d5b3ad8f76f24b6a2 Oct 30 22:15:03.168035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.167982 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:03.275596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.275555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:03.449628 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:03.448647 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.269s" Oct 30 22:15:03.449628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.448719 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:03.455746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.455366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerStarted","Data":"98c035d8ffc56f7a20b90c049f887681a2cc9d211cdf48aefab006a214809051"} Oct 30 22:15:03.467683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.463903 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerStarted","Data":"7a8ce4a7faf5409c106bf9e6777d5541005fd82b36d92975bc7c66f1b28b98c4"} Oct 30 22:15:03.493837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.484689 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84"] Oct 30 22:15:03.493837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.484734 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerStarted","Data":"da3b99e9a9ee748ca775e49ab2d70e5165cb5b60e6d9cb1d12b13596c748060e"} Oct 30 22:15:03.493837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.492576 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeec18-ppm84"] Oct 30 22:15:03.505960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.505783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerStarted","Data":"6318f8b0f5734e4988acd01b0f78a15179d0cc1501aff082fb7441e4fc0e2d63"} Oct 30 22:15:03.519999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.519134 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerStarted","Data":"6585a34f2e9dcba5afed4bbe7b5b24d7f8150b64a7df3e974cc56c48db2c7051"} Oct 30 22:15:03.519999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.519422 5036 scope.go:122] "RemoveContainer" containerID="6585a34f2e9dcba5afed4bbe7b5b24d7f8150b64a7df3e974cc56c48db2c7051" Oct 30 22:15:03.524235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.523044 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerStarted","Data":"8b207bac9199e8f36bbfb67cd94b6ec553039640cfce777c5a55f6dde974542a"} Oct 30 22:15:03.530254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.530176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerStarted","Data":"281dd6d624e68501dc7f1eee513db2ab555bc1553cbb63f51ae469815575a2d4"} Oct 30 22:15:03.537440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.537323 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerStarted","Data":"8894c35f06436e018397c7f76f7ec1470329244af88dc80cfcefe36d4093b8ac"} Oct 30 22:15:03.540550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.540521 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerStarted","Data":"322273b5993e787f26df3178a4bfbb4ed7cd5ebe253c06304c879a5cdf4b8280"} Oct 30 22:15:03.559573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.559409 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerStarted","Data":"abe1e6c826de10c06a98579f00c35cf679fd4eb85585d809181d5403a348192d"} Oct 30 22:15:03.563737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.563703 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerStarted","Data":"8210e2e19963a32e67762d5d4bd9725870993489a66e2ce7dbdcaf83870e7071"} Oct 30 22:15:03.569085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.569040 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerStarted","Data":"6d8abf7f9fecc54f493b52bb038179aa68ecd4eafde98a9fafb9e5afbe895019"} Oct 30 22:15:03.571516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.571492 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerStarted","Data":"2447be663b64590b840fb7220572c1c90d1faa9b76032e4d5b3ad8f76f24b6a2"} Oct 30 22:15:03.574036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.574018 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerStarted","Data":"612680b46f103ddcbd9b9676e383143ebfb7f7d789f7606a846a049cc050f6dc"} Oct 30 22:15:03.582357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.582310 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" podStartSLOduration=3.582295193 podStartE2EDuration="3.582295193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.579611235 +0000 UTC m=+2127.529566182" watchObservedRunningTime="2026-10-30 22:15:03.582295193 +0000 UTC m=+2127.532250146" Oct 30 22:15:03.620741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.620685 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" podStartSLOduration=3.6206642799999997 podStartE2EDuration="3.62066428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.599720903 +0000 UTC m=+2127.549675850" watchObservedRunningTime="2026-10-30 22:15:03.62066428 +0000 UTC m=+2127.570619227" Oct 30 22:15:03.653494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:03.653441 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" podStartSLOduration=3.653419605 podStartE2EDuration="3.653419605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.643325415 +0000 UTC m=+2127.593280709" watchObservedRunningTime="2026-10-30 22:15:03.653419605 +0000 UTC m=+2127.603374552" Oct 30 22:15:04.348441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.343320 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22204fc-bb5b-4eec-bb15-4e83ae18078a" path="/var/lib/kubelet/pods/d22204fc-bb5b-4eec-bb15-4e83ae18078a/volumes" Oct 30 22:15:04.733301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.732697 5036 generic.go:356] "Generic (PLEG): container finished" podID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerID="2312c2c937a9565578cd5e1e9b4004c6e30137ae5035f3fa0a5738b3ac5ddbd1" exitCode=60 Oct 30 22:15:04.734948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.733645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerDied","Data":"2312c2c937a9565578cd5e1e9b4004c6e30137ae5035f3fa0a5738b3ac5ddbd1"} Oct 30 22:15:04.734948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.734176 5036 scope.go:122] "RemoveContainer" containerID="2312c2c937a9565578cd5e1e9b4004c6e30137ae5035f3fa0a5738b3ac5ddbd1" Oct 30 22:15:04.769800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.768340 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerID="f8c3b4fd6ad04ad4386846e66eb0236e191686af6685169d0a8267413d8dbeb8" exitCode=60 Oct 30 22:15:04.769800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.768489 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerDied","Data":"f8c3b4fd6ad04ad4386846e66eb0236e191686af6685169d0a8267413d8dbeb8"} Oct 30 22:15:04.769800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.768925 5036 scope.go:122] "RemoveContainer" containerID="f8c3b4fd6ad04ad4386846e66eb0236e191686af6685169d0a8267413d8dbeb8" Oct 30 22:15:04.791222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.789502 5036 generic.go:356] "Generic (PLEG): container finished" podID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerID="6d71d6d89fb90c2a5eb1c6427ccca6d78ba7c4da925c45bef93b1f7d0be9a659" exitCode=60 Oct 30 22:15:04.791222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.789605 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerDied","Data":"6d71d6d89fb90c2a5eb1c6427ccca6d78ba7c4da925c45bef93b1f7d0be9a659"} Oct 30 22:15:04.791222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.790084 5036 scope.go:122] "RemoveContainer" containerID="6d71d6d89fb90c2a5eb1c6427ccca6d78ba7c4da925c45bef93b1f7d0be9a659" Oct 30 22:15:04.804797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.804537 5036 generic.go:356] "Generic (PLEG): container finished" podID="48f7a48b-c603-45e3-a20b-9630c280f391" containerID="6585a34f2e9dcba5afed4bbe7b5b24d7f8150b64a7df3e974cc56c48db2c7051" exitCode=60 Oct 30 22:15:04.804797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.804703 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerDied","Data":"6585a34f2e9dcba5afed4bbe7b5b24d7f8150b64a7df3e974cc56c48db2c7051"} Oct 30 22:15:04.823833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.822502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerStarted","Data":"16b4f2e90a0afd62694109a80e21f8f072688750937a2f10c1aa69a415289860"} Oct 30 22:15:04.823833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.823044 5036 scope.go:122] "RemoveContainer" containerID="16b4f2e90a0afd62694109a80e21f8f072688750937a2f10c1aa69a415289860" Oct 30 22:15:04.844512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.844473 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerID="a36bdf150c8e67f5066bd7421fdee82e93b69935b131094bdc10864dae89972b" exitCode=60 Oct 30 22:15:04.844641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.844590 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerDied","Data":"a36bdf150c8e67f5066bd7421fdee82e93b69935b131094bdc10864dae89972b"} Oct 30 22:15:04.845676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.845099 5036 scope.go:122] "RemoveContainer" containerID="a36bdf150c8e67f5066bd7421fdee82e93b69935b131094bdc10864dae89972b" Oct 30 22:15:04.849555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.849523 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerStarted","Data":"da68b6b3209de28a65f63c1fdc1d1699b901d0df658edde419fcdc83cfa062b2"} Oct 30 22:15:04.850013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.849988 5036 scope.go:122] "RemoveContainer" containerID="da68b6b3209de28a65f63c1fdc1d1699b901d0df658edde419fcdc83cfa062b2" Oct 30 22:15:04.853249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.853219 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerStarted","Data":"57971e6026233a9485768ebb7d4ff8548dbb569c3aa463c19ec970ff0ac67e1c"} Oct 30 22:15:04.853831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.853807 5036 scope.go:122] "RemoveContainer" containerID="57971e6026233a9485768ebb7d4ff8548dbb569c3aa463c19ec970ff0ac67e1c" Oct 30 22:15:04.877687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:04.874370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerStarted","Data":"c2b5a829408a320768c31524e01dba2bbab8cfa442cd59b8d027e3a489740f66"} Oct 30 22:15:05.006591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.006538 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" podStartSLOduration=5.006519951 podStartE2EDuration="5.006519951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.99873437 +0000 UTC m=+2128.948689320" watchObservedRunningTime="2026-10-30 22:15:05.006519951 +0000 UTC m=+2128.956474901" Oct 30 22:15:05.117506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.117440 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" podStartSLOduration=5.117418423 podStartE2EDuration="5.117418423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.064370725 +0000 UTC m=+2129.014325671" watchObservedRunningTime="2026-10-30 22:15:05.117418423 +0000 UTC m=+2129.067373370" Oct 30 22:15:05.931289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.930426 5036 generic.go:356] "Generic (PLEG): container finished" podID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerID="ce6b551aca82e5fa1c6fd4b054bdc9f9f61f40669749dfd97cfee1295c624c89" exitCode=60 Oct 30 22:15:05.931289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.930575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerDied","Data":"ce6b551aca82e5fa1c6fd4b054bdc9f9f61f40669749dfd97cfee1295c624c89"} Oct 30 22:15:05.931785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.931760 5036 scope.go:122] "RemoveContainer" containerID="ce6b551aca82e5fa1c6fd4b054bdc9f9f61f40669749dfd97cfee1295c624c89" Oct 30 22:15:05.940505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.936917 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerStarted","Data":"79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0"} Oct 30 22:15:05.943658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.943215 5036 scope.go:122] "RemoveContainer" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" Oct 30 22:15:05.943658 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:05.943576 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6_odh-ai-gateway-infra(48f7a48b-c603-45e3-a20b-9630c280f391)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" Oct 30 22:15:05.944220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.944198 5036 generic.go:356] "Generic (PLEG): container finished" podID="48f7a48b-c603-45e3-a20b-9630c280f391" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" exitCode=60 Oct 30 22:15:05.944514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.944497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerDied","Data":"33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e"} Oct 30 22:15:05.944845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.944827 5036 scope.go:122] "RemoveContainer" containerID="6585a34f2e9dcba5afed4bbe7b5b24d7f8150b64a7df3e974cc56c48db2c7051" Oct 30 22:15:05.948892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.948867 5036 generic.go:356] "Generic (PLEG): container finished" podID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerID="16b4f2e90a0afd62694109a80e21f8f072688750937a2f10c1aa69a415289860" exitCode=60 Oct 30 22:15:05.949009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.948928 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerDied","Data":"16b4f2e90a0afd62694109a80e21f8f072688750937a2f10c1aa69a415289860"} Oct 30 22:15:05.951963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.951943 5036 generic.go:356] "Generic (PLEG): container finished" podID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerID="da68b6b3209de28a65f63c1fdc1d1699b901d0df658edde419fcdc83cfa062b2" exitCode=60 Oct 30 22:15:05.952057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.952016 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerDied","Data":"da68b6b3209de28a65f63c1fdc1d1699b901d0df658edde419fcdc83cfa062b2"} Oct 30 22:15:05.954215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.954186 5036 generic.go:356] "Generic (PLEG): container finished" podID="7dd08765-1f01-4991-beef-acae8a938a12" containerID="e088fed03d18b75e4d8f4cb421c6f8df354ddfdfd3acc209889c6c1ed1406176" exitCode=60 Oct 30 22:15:05.954335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.954322 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerDied","Data":"e088fed03d18b75e4d8f4cb421c6f8df354ddfdfd3acc209889c6c1ed1406176"} Oct 30 22:15:05.954595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.954576 5036 scope.go:122] "RemoveContainer" containerID="e088fed03d18b75e4d8f4cb421c6f8df354ddfdfd3acc209889c6c1ed1406176" Oct 30 22:15:05.957183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.957105 5036 generic.go:356] "Generic (PLEG): container finished" podID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerID="57971e6026233a9485768ebb7d4ff8548dbb569c3aa463c19ec970ff0ac67e1c" exitCode=60 Oct 30 22:15:05.957183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.957172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerDied","Data":"57971e6026233a9485768ebb7d4ff8548dbb569c3aa463c19ec970ff0ac67e1c"} Oct 30 22:15:05.959922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.959894 5036 generic.go:356] "Generic (PLEG): container finished" podID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerID="ae2b9e58990970e63efcdb2d3802dda41e68728804a5bf363ee73d0f9673d318" exitCode=60 Oct 30 22:15:05.959993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.959975 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerDied","Data":"ae2b9e58990970e63efcdb2d3802dda41e68728804a5bf363ee73d0f9673d318"} Oct 30 22:15:05.960347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.960203 5036 scope.go:122] "RemoveContainer" containerID="ae2b9e58990970e63efcdb2d3802dda41e68728804a5bf363ee73d0f9673d318" Oct 30 22:15:05.963828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.963802 5036 generic.go:356] "Generic (PLEG): container finished" podID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerID="411c5c3afc832e7bef5e0dfc6acf6b3e6bbdc2329f335e67d68169292d0cc89d" exitCode=60 Oct 30 22:15:05.963950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.963899 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerDied","Data":"411c5c3afc832e7bef5e0dfc6acf6b3e6bbdc2329f335e67d68169292d0cc89d"} Oct 30 22:15:05.964361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.964326 5036 scope.go:122] "RemoveContainer" containerID="411c5c3afc832e7bef5e0dfc6acf6b3e6bbdc2329f335e67d68169292d0cc89d" Oct 30 22:15:05.967361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.967339 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerStarted","Data":"21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91"} Oct 30 22:15:05.969906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.969887 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerID="ccd6b460fd324867411528303c54c076c430ea3fba8d65c62e1d900978dd78a7" exitCode=60 Oct 30 22:15:05.970104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.970004 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerDied","Data":"ccd6b460fd324867411528303c54c076c430ea3fba8d65c62e1d900978dd78a7"} Oct 30 22:15:05.970222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.970201 5036 scope.go:122] "RemoveContainer" containerID="ccd6b460fd324867411528303c54c076c430ea3fba8d65c62e1d900978dd78a7" Oct 30 22:15:05.976709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.976681 5036 generic.go:356] "Generic (PLEG): container finished" podID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerID="0ad767067bc123de96e0ab974f51d22580d3fdc46b161d82b7d8faab78c86f68" exitCode=60 Oct 30 22:15:05.976841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.976808 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerDied","Data":"0ad767067bc123de96e0ab974f51d22580d3fdc46b161d82b7d8faab78c86f68"} Oct 30 22:15:05.977110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.977092 5036 scope.go:122] "RemoveContainer" containerID="0ad767067bc123de96e0ab974f51d22580d3fdc46b161d82b7d8faab78c86f68" Oct 30 22:15:05.992739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.991502 5036 generic.go:356] "Generic (PLEG): container finished" podID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerID="1b30bfdf3ed2c4aad50f13a0d5478ccd6d039717c60e9e7448d05a424438cabd" exitCode=60 Oct 30 22:15:05.992739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.991576 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerDied","Data":"1b30bfdf3ed2c4aad50f13a0d5478ccd6d039717c60e9e7448d05a424438cabd"} Oct 30 22:15:05.992739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.991927 5036 scope.go:122] "RemoveContainer" containerID="1b30bfdf3ed2c4aad50f13a0d5478ccd6d039717c60e9e7448d05a424438cabd" Oct 30 22:15:05.993694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:05.992355 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" podStartSLOduration=5.992341473 podStartE2EDuration="5.992341473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.9858149 +0000 UTC m=+2129.935769845" watchObservedRunningTime="2026-10-30 22:15:05.992341473 +0000 UTC m=+2129.942296420" Oct 30 22:15:06.005827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.004350 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerStarted","Data":"7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9"} Oct 30 22:15:06.181355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.181294 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" podStartSLOduration=6.181258625 podStartE2EDuration="6.181258625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:06.178695547 +0000 UTC m=+2130.128650494" watchObservedRunningTime="2026-10-30 22:15:06.181258625 +0000 UTC m=+2130.131213573" Oct 30 22:15:06.238958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.238912 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" podUID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" containerName="payload-processing" containerID="cri-o://2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613" gracePeriod=25 Oct 30 22:15:06.814717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.813961 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:15:06.880297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.879808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\") pod \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " Oct 30 22:15:06.880297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.879868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\") pod \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " Oct 30 22:15:06.880297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.879932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\") pod \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " Oct 30 22:15:06.880297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.880085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\") pod \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\" (UID: \"6681db39-45c7-45ea-85ce-d4c6f64daa2d\") " Oct 30 22:15:06.896545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.885623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume" pod "6681db39-45c7-45ea-85ce-d4c6f64daa2d" (UID: "6681db39-45c7-45ea-85ce-d4c6f64daa2d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:06.896545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.885864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume" pod "6681db39-45c7-45ea-85ce-d4c6f64daa2d" (UID: "6681db39-45c7-45ea-85ce-d4c6f64daa2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:06.910049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.909979 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665" pod "6681db39-45c7-45ea-85ce-d4c6f64daa2d" (UID: "6681db39-45c7-45ea-85ce-d4c6f64daa2d"). InnerVolumeSpecName "kube-api-access-vx665". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:06.910542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:06.910491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials" pod "6681db39-45c7-45ea-85ce-d4c6f64daa2d" (UID: "6681db39-45c7-45ea-85ce-d4c6f64daa2d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:07.047945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.039483 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:07.047945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.039519 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:07.047945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.039537 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6681db39-45c7-45ea-85ce-d4c6f64daa2d-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:07.047945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.039573 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx665\" (UniqueName: \"kubernetes.io/projected/6681db39-45c7-45ea-85ce-d4c6f64daa2d-kube-api-access-vx665\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:07.102938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.102907 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerStarted","Data":"2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d"} Oct 30 22:15:07.105893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.105859 5036 generic.go:356] "Generic (PLEG): container finished" podID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerID="21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91" exitCode=60 Oct 30 22:15:07.106054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.105946 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerDied","Data":"21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91"} Oct 30 22:15:07.106054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.106017 5036 scope.go:122] "RemoveContainer" containerID="2312c2c937a9565578cd5e1e9b4004c6e30137ae5035f3fa0a5738b3ac5ddbd1" Oct 30 22:15:07.106248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.106227 5036 scope.go:122] "RemoveContainer" containerID="21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91" Oct 30 22:15:07.106710 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.106682 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw_odh-ai-gateway-infra(b6bd0e01-0001-426e-b7a3-e8140661136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" Oct 30 22:15:07.110142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.110117 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerStarted","Data":"9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92"} Oct 30 22:15:07.114936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.114910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerStarted","Data":"3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee"} Oct 30 22:15:07.118574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.118552 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerStarted","Data":"61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e"} Oct 30 22:15:07.121010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.120988 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerID="7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9" exitCode=60 Oct 30 22:15:07.121123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.121079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerDied","Data":"7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9"} Oct 30 22:15:07.123566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.123452 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerStarted","Data":"b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40"} Oct 30 22:15:07.123678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.123574 5036 scope.go:122] "RemoveContainer" containerID="7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9" Oct 30 22:15:07.124383 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.123891 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz_odh-ai-gateway-infra(8c761ec4-d3b0-4663-88e7-2de45be4f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" Oct 30 22:15:07.126680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.126650 5036 generic.go:356] "Generic (PLEG): container finished" podID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerID="79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0" exitCode=60 Oct 30 22:15:07.126787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.126762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerDied","Data":"79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0"} Oct 30 22:15:07.126989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.126969 5036 scope.go:122] "RemoveContainer" containerID="79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0" Oct 30 22:15:07.127557 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.127531 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv_odh-ai-gateway-infra(e6802e98-f7d3-4b61-9d03-ddec2b4442fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" Oct 30 22:15:07.128881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.128860 5036 scope.go:122] "RemoveContainer" containerID="f8c3b4fd6ad04ad4386846e66eb0236e191686af6685169d0a8267413d8dbeb8" Oct 30 22:15:07.129407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.129382 5036 scope.go:122] "RemoveContainer" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" Oct 30 22:15:07.130426 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.130056 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6_odh-ai-gateway-infra(48f7a48b-c603-45e3-a20b-9630c280f391)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" Oct 30 22:15:07.132748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.132727 5036 generic.go:356] "Generic (PLEG): container finished" podID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" containerID="2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613" exitCode=0 Oct 30 22:15:07.132857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.132759 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" Oct 30 22:15:07.132857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.132799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" event={"ID":"6681db39-45c7-45ea-85ce-d4c6f64daa2d","Type":"ContainerDied","Data":"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613"} Oct 30 22:15:07.132857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.132827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb" event={"ID":"6681db39-45c7-45ea-85ce-d4c6f64daa2d","Type":"ContainerDied","Data":"80941c84f796c4552f19a2f66f9e10a7d45d03f2e9e387b57218f6600f35ba41"} Oct 30 22:15:07.135239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.135066 5036 generic.go:356] "Generic (PLEG): container finished" podID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" exitCode=60 Oct 30 22:15:07.136026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.136007 5036 scope.go:122] "RemoveContainer" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" Oct 30 22:15:07.136244 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.136224 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57_odh-ai-gateway-infra(daf049fb-3442-41b6-abe4-b43daeaf22e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" Oct 30 22:15:07.136925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.136898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerDied","Data":"1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057"} Oct 30 22:15:07.139302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.137778 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerID="008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb" exitCode=60 Oct 30 22:15:07.139302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.137890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerDied","Data":"008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb"} Oct 30 22:15:07.139302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.138815 5036 scope.go:122] "RemoveContainer" containerID="008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb" Oct 30 22:15:07.139302 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.139203 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh_odh-ai-gateway-infra(7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" Oct 30 22:15:07.144052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.144028 5036 generic.go:356] "Generic (PLEG): container finished" podID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerID="09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978" exitCode=60 Oct 30 22:15:07.144277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.144242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerDied","Data":"09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978"} Oct 30 22:15:07.144479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.144457 5036 scope.go:122] "RemoveContainer" containerID="09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978" Oct 30 22:15:07.144957 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.144928 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b_odh-ai-gateway-infra(baf9a179-a462-4c72-aaf5-cd1af5717093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" Oct 30 22:15:07.147701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.147674 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerStarted","Data":"2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd"} Oct 30 22:15:07.151975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.151427 5036 scope.go:122] "RemoveContainer" containerID="6d71d6d89fb90c2a5eb1c6427ccca6d78ba7c4da925c45bef93b1f7d0be9a659" Oct 30 22:15:07.152770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.152746 5036 generic.go:356] "Generic (PLEG): container finished" podID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerID="c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1" exitCode=60 Oct 30 22:15:07.152889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.152835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerDied","Data":"c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1"} Oct 30 22:15:07.153366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.153345 5036 scope.go:122] "RemoveContainer" containerID="c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1" Oct 30 22:15:07.153660 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.153639 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s_odh-ai-gateway-infra(8527b219-c5dc-45ae-b088-0db98e8e1042)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" Oct 30 22:15:07.155660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.155635 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerStarted","Data":"9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952"} Oct 30 22:15:07.169569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.169529 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" podStartSLOduration=7.1695157720000005 podStartE2EDuration="7.169515772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.166498103 +0000 UTC m=+2131.116453049" watchObservedRunningTime="2026-10-30 22:15:07.169515772 +0000 UTC m=+2131.119470724" Oct 30 22:15:07.171176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.171062 5036 scope.go:122] "RemoveContainer" containerID="2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613" Oct 30 22:15:07.185053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.185027 5036 scope.go:122] "RemoveContainer" containerID="2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613" Oct 30 22:15:07.185379 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:07.185359 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613\": container with ID starting with 2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613 not found: ID does not exist" containerID="2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613" Oct 30 22:15:07.185454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.185388 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613"} err="failed to get container status \"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613\": rpc error: code = NotFound desc = could not find container \"2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613\": container with ID starting with 2fa4c5314c4c38642b5085940ab484cb1b433af00b7672bc8efae026a1702613 not found: ID does not exist" Oct 30 22:15:07.185454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.185405 5036 scope.go:122] "RemoveContainer" containerID="16b4f2e90a0afd62694109a80e21f8f072688750937a2f10c1aa69a415289860" Oct 30 22:15:07.197780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.197740 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" podStartSLOduration=7.197727036 podStartE2EDuration="7.197727036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.192036213 +0000 UTC m=+2131.141991161" watchObservedRunningTime="2026-10-30 22:15:07.197727036 +0000 UTC m=+2131.147681987" Oct 30 22:15:07.211934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.211816 5036 scope.go:122] "RemoveContainer" containerID="a36bdf150c8e67f5066bd7421fdee82e93b69935b131094bdc10864dae89972b" Oct 30 22:15:07.217373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.217329 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" podStartSLOduration=7.21731215 podStartE2EDuration="7.21731215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.213339009 +0000 UTC m=+2131.163293957" watchObservedRunningTime="2026-10-30 22:15:07.21731215 +0000 UTC m=+2131.167267104" Oct 30 22:15:07.233550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.233518 5036 scope.go:122] "RemoveContainer" containerID="da68b6b3209de28a65f63c1fdc1d1699b901d0df658edde419fcdc83cfa062b2" Oct 30 22:15:07.250999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.250978 5036 scope.go:122] "RemoveContainer" containerID="57971e6026233a9485768ebb7d4ff8548dbb569c3aa463c19ec970ff0ac67e1c" Oct 30 22:15:07.265330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.265241 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" podStartSLOduration=7.265223333 podStartE2EDuration="7.265223333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.263590846 +0000 UTC m=+2131.213545798" watchObservedRunningTime="2026-10-30 22:15:07.265223333 +0000 UTC m=+2131.215178286" Oct 30 22:15:07.282412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.282383 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:15:07.288710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.288685 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-77585kvlb"] Oct 30 22:15:07.373065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.372956 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" podStartSLOduration=7.372929722 podStartE2EDuration="7.372929722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.368993088 +0000 UTC m=+2131.318948037" watchObservedRunningTime="2026-10-30 22:15:07.372929722 +0000 UTC m=+2131.322884667" Oct 30 22:15:07.386707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.386680 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx"] Oct 30 22:15:07.387703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.387683 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" containerName="payload-processing" Oct 30 22:15:07.387703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.387703 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" containerName="payload-processing" Oct 30 22:15:07.387902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.387876 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" containerName="payload-processing" Oct 30 22:15:07.393545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.393526 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.404667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.403664 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:07.465653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.465594 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbf6\" (UniqueName: \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.466051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.466027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.466413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.466390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.486468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.486408 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx"] Oct 30 22:15:07.567958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.567925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.568072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.568037 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.568343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.568321 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbf6\" (UniqueName: \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.569098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.569070 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.570579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.570558 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.583901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.583878 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbf6\" (UniqueName: \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\") pod \"maas-api-revoke-sub-9831cd94-q69hx\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:07.750466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:07.742092 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:08.180400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.179436 5036 scope.go:122] "RemoveContainer" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" Oct 30 22:15:08.180400 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.179960 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57_odh-ai-gateway-infra(daf049fb-3442-41b6-abe4-b43daeaf22e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" Oct 30 22:15:08.192214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.191621 5036 scope.go:122] "RemoveContainer" containerID="008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb" Oct 30 22:15:08.192214 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.191960 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh_odh-ai-gateway-infra(7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" Oct 30 22:15:08.201288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.198169 5036 scope.go:122] "RemoveContainer" containerID="09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978" Oct 30 22:15:08.201288 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.198530 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b_odh-ai-gateway-infra(baf9a179-a462-4c72-aaf5-cd1af5717093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" Oct 30 22:15:08.203408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.203379 5036 generic.go:356] "Generic (PLEG): container finished" podID="7dd08765-1f01-4991-beef-acae8a938a12" containerID="2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd" exitCode=60 Oct 30 22:15:08.203789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.203768 5036 scope.go:122] "RemoveContainer" containerID="2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd" Oct 30 22:15:08.204073 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.204047 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj_odh-ai-gateway-infra(7dd08765-1f01-4991-beef-acae8a938a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" podUID="7dd08765-1f01-4991-beef-acae8a938a12" Oct 30 22:15:08.213176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.213146 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6681db39-45c7-45ea-85ce-d4c6f64daa2d" path="/var/lib/kubelet/pods/6681db39-45c7-45ea-85ce-d4c6f64daa2d/volumes" Oct 30 22:15:08.213716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.213690 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerDied","Data":"2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd"} Oct 30 22:15:08.213827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.213815 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:15:08.214140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.214117 5036 scope.go:122] "RemoveContainer" containerID="e088fed03d18b75e4d8f4cb421c6f8df354ddfdfd3acc209889c6c1ed1406176" Oct 30 22:15:08.214595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.214572 5036 scope.go:122] "RemoveContainer" containerID="c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1" Oct 30 22:15:08.214902 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.214873 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s_odh-ai-gateway-infra(8527b219-c5dc-45ae-b088-0db98e8e1042)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" Oct 30 22:15:08.218570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.218547 5036 generic.go:356] "Generic (PLEG): container finished" podID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerID="9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952" exitCode=60 Oct 30 22:15:08.219200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.219031 5036 scope.go:122] "RemoveContainer" containerID="9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952" Oct 30 22:15:08.219613 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.219589 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz_odh-ai-gateway-infra(93159d6d-45b9-4b2c-8bcd-4dd2d3507b50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" Oct 30 22:15:08.220101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.219376 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerDied","Data":"9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952"} Oct 30 22:15:08.223127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.223105 5036 generic.go:356] "Generic (PLEG): container finished" podID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerID="2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d" exitCode=60 Oct 30 22:15:08.223389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.223180 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerDied","Data":"2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d"} Oct 30 22:15:08.223976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.223938 5036 scope.go:122] "RemoveContainer" containerID="2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d" Oct 30 22:15:08.224698 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.224677 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f_odh-ai-gateway-infra(9ca06018-5e9c-4cc9-b360-9ead31686bf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" Oct 30 22:15:08.229420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.229082 5036 scope.go:122] "RemoveContainer" containerID="21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91" Oct 30 22:15:08.229420 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.229388 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw_odh-ai-gateway-infra(b6bd0e01-0001-426e-b7a3-e8140661136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" Oct 30 22:15:08.230431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.230414 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:15:08.232279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.232246 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerID="9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92" exitCode=60 Oct 30 22:15:08.233002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.232382 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerDied","Data":"9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92"} Oct 30 22:15:08.233002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.232608 5036 scope.go:122] "RemoveContainer" containerID="9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92" Oct 30 22:15:08.233002 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.232966 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf_odh-ai-gateway-infra(cd97a033-6160-4dc2-bde2-0ab637df4548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" Oct 30 22:15:08.237155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.236974 5036 generic.go:356] "Generic (PLEG): container finished" podID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerID="3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee" exitCode=60 Oct 30 22:15:08.237155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.237041 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerDied","Data":"3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee"} Oct 30 22:15:08.237358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.237295 5036 scope.go:122] "RemoveContainer" containerID="3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee" Oct 30 22:15:08.237638 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.237609 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-lqx56_odh-ai-gateway-infra(13459fce-bfbc-45e0-8e1a-7c6b567c76e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" Oct 30 22:15:08.241310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.241242 5036 generic.go:356] "Generic (PLEG): container finished" podID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerID="61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e" exitCode=60 Oct 30 22:15:08.241425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.241336 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerDied","Data":"61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e"} Oct 30 22:15:08.241701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.241679 5036 scope.go:122] "RemoveContainer" containerID="61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e" Oct 30 22:15:08.242006 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.241960 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq_odh-ai-gateway-infra(e48a647e-1b9e-4248-aa22-1349c838eb04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" Oct 30 22:15:08.247185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.247167 5036 scope.go:122] "RemoveContainer" containerID="7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9" Oct 30 22:15:08.247642 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.247616 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz_odh-ai-gateway-infra(8c761ec4-d3b0-4663-88e7-2de45be4f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" Oct 30 22:15:08.261228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.260145 5036 scope.go:122] "RemoveContainer" containerID="ae2b9e58990970e63efcdb2d3802dda41e68728804a5bf363ee73d0f9673d318" Oct 30 22:15:08.264149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.261533 5036 generic.go:356] "Generic (PLEG): container finished" podID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerID="b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40" exitCode=60 Oct 30 22:15:08.264149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.261598 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerDied","Data":"b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40"} Oct 30 22:15:08.264149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.261989 5036 scope.go:122] "RemoveContainer" containerID="b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40" Oct 30 22:15:08.264149 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.262318 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2_odh-ai-gateway-infra(2e4148eb-52e6-4c5b-ba64-3948bc5533ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" Oct 30 22:15:08.276672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.276645 5036 scope.go:122] "RemoveContainer" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" Oct 30 22:15:08.277173 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.277143 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6_odh-ai-gateway-infra(48f7a48b-c603-45e3-a20b-9630c280f391)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" Oct 30 22:15:08.277715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.277695 5036 scope.go:122] "RemoveContainer" containerID="79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0" Oct 30 22:15:08.278140 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:08.278116 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv_odh-ai-gateway-infra(e6802e98-f7d3-4b61-9d03-ddec2b4442fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" Oct 30 22:15:08.296380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.296353 5036 scope.go:122] "RemoveContainer" containerID="411c5c3afc832e7bef5e0dfc6acf6b3e6bbdc2329f335e67d68169292d0cc89d" Oct 30 22:15:08.313179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.313158 5036 scope.go:122] "RemoveContainer" containerID="ccd6b460fd324867411528303c54c076c430ea3fba8d65c62e1d900978dd78a7" Oct 30 22:15:08.369884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.369311 5036 scope.go:122] "RemoveContainer" containerID="0ad767067bc123de96e0ab974f51d22580d3fdc46b161d82b7d8faab78c86f68" Oct 30 22:15:08.375709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.370470 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx"] Oct 30 22:15:08.441003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.440982 5036 scope.go:122] "RemoveContainer" containerID="1b30bfdf3ed2c4aad50f13a0d5478ccd6d039717c60e9e7448d05a424438cabd" Oct 30 22:15:08.488996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:08.488969 5036 scope.go:122] "RemoveContainer" containerID="ce6b551aca82e5fa1c6fd4b054bdc9f9f61f40669749dfd97cfee1295c624c89" Oct 30 22:15:09.290905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.290880 5036 scope.go:122] "RemoveContainer" containerID="b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40" Oct 30 22:15:09.291428 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.291335 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2_odh-ai-gateway-infra(2e4148eb-52e6-4c5b-ba64-3948bc5533ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" Oct 30 22:15:09.292205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.292171 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" event={"ID":"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0","Type":"ContainerStarted","Data":"b53622715a967bdb77e6b8df3ff35ab91f0145ca6f27493bd52e648d1a9ce663"} Oct 30 22:15:09.292377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.292279 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" event={"ID":"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0","Type":"ContainerStarted","Data":"2c0b467ce51218a43531d7029c7c9dd446588f70b30481cef45f5957596061f5"} Oct 30 22:15:09.294354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.294331 5036 scope.go:122] "RemoveContainer" containerID="2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd" Oct 30 22:15:09.294722 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.294695 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj_odh-ai-gateway-infra(7dd08765-1f01-4991-beef-acae8a938a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" podUID="7dd08765-1f01-4991-beef-acae8a938a12" Oct 30 22:15:09.296277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.296241 5036 scope.go:122] "RemoveContainer" containerID="9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952" Oct 30 22:15:09.296656 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.296627 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz_odh-ai-gateway-infra(93159d6d-45b9-4b2c-8bcd-4dd2d3507b50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" Oct 30 22:15:09.299763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.299744 5036 scope.go:122] "RemoveContainer" containerID="9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92" Oct 30 22:15:09.300098 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.300071 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf_odh-ai-gateway-infra(cd97a033-6160-4dc2-bde2-0ab637df4548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" Oct 30 22:15:09.301468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.301435 5036 scope.go:122] "RemoveContainer" containerID="3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee" Oct 30 22:15:09.301776 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.301753 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-lqx56_odh-ai-gateway-infra(13459fce-bfbc-45e0-8e1a-7c6b567c76e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" Oct 30 22:15:09.303104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.303083 5036 scope.go:122] "RemoveContainer" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" Oct 30 22:15:09.303194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.303162 5036 scope.go:122] "RemoveContainer" containerID="61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e" Oct 30 22:15:09.303447 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.303417 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq_odh-ai-gateway-infra(e48a647e-1b9e-4248-aa22-1349c838eb04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" Oct 30 22:15:09.303497 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:09.303417 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57_odh-ai-gateway-infra(daf049fb-3442-41b6-abe4-b43daeaf22e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" Oct 30 22:15:09.363889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:09.363800 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" podStartSLOduration=2.363779777 podStartE2EDuration="2.363779777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:09.363482747 +0000 UTC m=+2133.313437694" watchObservedRunningTime="2026-10-30 22:15:09.363779777 +0000 UTC m=+2133.313734724" Oct 30 22:15:10.316254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:10.316218 5036 generic.go:356] "Generic (PLEG): container finished" podID="76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" containerID="b53622715a967bdb77e6b8df3ff35ab91f0145ca6f27493bd52e648d1a9ce663" exitCode=0 Oct 30 22:15:10.316776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:10.316316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" event={"ID":"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0","Type":"ContainerDied","Data":"b53622715a967bdb77e6b8df3ff35ab91f0145ca6f27493bd52e648d1a9ce663"} Oct 30 22:15:11.557714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.557683 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:11.690950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.690910 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\") pod \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " Oct 30 22:15:11.691141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.691037 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\" (UniqueName: \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\") pod \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " Oct 30 22:15:11.691141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.691099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\") pod \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\" (UID: \"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0\") " Oct 30 22:15:11.691493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.691459 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca" pod "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" (UID: "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:11.691812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.691794 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:11.693399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.693367 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca" pod "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" (UID: "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:11.693513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.693491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6" pod "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" (UID: "76ab6290-d5d2-4a45-84cf-13ae2a85c6a0"). InnerVolumeSpecName "kube-api-access-tpbf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:11.793483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.793450 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:11.793483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:11.793478 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpbf6\" (UniqueName: \"kubernetes.io/projected/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0-kube-api-access-tpbf6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:12.344667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:12.344636 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" Oct 30 22:15:12.344859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:12.344668 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx" event={"ID":"76ab6290-d5d2-4a45-84cf-13ae2a85c6a0","Type":"ContainerDied","Data":"2c0b467ce51218a43531d7029c7c9dd446588f70b30481cef45f5957596061f5"} Oct 30 22:15:12.344859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:12.344731 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0b467ce51218a43531d7029c7c9dd446588f70b30481cef45f5957596061f5" Oct 30 22:15:13.048056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:13.048016 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x"] Oct 30 22:15:13.057286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:13.055287 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87235e6d-ztd5x"] Oct 30 22:15:14.191962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.191923 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c17be6f-6ca1-4625-a31a-524c21682cfb" path="/var/lib/kubelet/pods/4c17be6f-6ca1-4625-a31a-524c21682cfb/volumes" Oct 30 22:15:14.304024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.303982 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" podUID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" containerName="payload-processing" containerID="cri-o://b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a" gracePeriod=25 Oct 30 22:15:14.632466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.632439 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:15:14.670067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.670030 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\") pod \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " Oct 30 22:15:14.670339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.670205 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\") pod \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " Oct 30 22:15:14.670436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.670394 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\") pod \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " Oct 30 22:15:14.670640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.670613 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\") pod \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\" (UID: \"7f86e877-9b24-44fa-8607-1f6b75fc7f46\") " Oct 30 22:15:14.671291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.671251 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume" pod "7f86e877-9b24-44fa-8607-1f6b75fc7f46" (UID: "7f86e877-9b24-44fa-8607-1f6b75fc7f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:14.671610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.671542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume" pod "7f86e877-9b24-44fa-8607-1f6b75fc7f46" (UID: "7f86e877-9b24-44fa-8607-1f6b75fc7f46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:14.673929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.673889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl" pod "7f86e877-9b24-44fa-8607-1f6b75fc7f46" (UID: "7f86e877-9b24-44fa-8607-1f6b75fc7f46"). InnerVolumeSpecName "kube-api-access-dxldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:14.674819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.674792 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials" pod "7f86e877-9b24-44fa-8607-1f6b75fc7f46" (UID: "7f86e877-9b24-44fa-8607-1f6b75fc7f46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:14.777068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.776957 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:14.777068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.777004 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:14.777068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.777022 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxldl\" (UniqueName: \"kubernetes.io/projected/7f86e877-9b24-44fa-8607-1f6b75fc7f46-kube-api-access-dxldl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:14.777068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:14.777066 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f86e877-9b24-44fa-8607-1f6b75fc7f46-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:15.427486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.427454 5036 generic.go:356] "Generic (PLEG): container finished" podID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" containerID="b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a" exitCode=0 Oct 30 22:15:15.427953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.427502 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" Oct 30 22:15:15.427953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.427537 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" event={"ID":"7f86e877-9b24-44fa-8607-1f6b75fc7f46","Type":"ContainerDied","Data":"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a"} Oct 30 22:15:15.427953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.427600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6" event={"ID":"7f86e877-9b24-44fa-8607-1f6b75fc7f46","Type":"ContainerDied","Data":"dcdfe5dae0859a6da26859087eee511b460823b6374cbc3edbba2735784d66d0"} Oct 30 22:15:15.427953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.427630 5036 scope.go:122] "RemoveContainer" containerID="b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a" Oct 30 22:15:15.442277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.442241 5036 scope.go:122] "RemoveContainer" containerID="b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a" Oct 30 22:15:15.442644 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:15.442625 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a\": container with ID starting with b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a not found: ID does not exist" containerID="b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a" Oct 30 22:15:15.442722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.442654 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a"} err="failed to get container status \"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a\": rpc error: code = NotFound desc = could not find container \"b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a\": container with ID starting with b01d1fea1d4875a187ef608d6175ec754c5b61b1540b08b4691fecc39c1ced4a not found: ID does not exist" Oct 30 22:15:15.456760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.456737 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:15:15.463082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:15.463056 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8hwzr6"] Oct 30 22:15:16.191818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:16.191787 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" path="/var/lib/kubelet/pods/7f86e877-9b24-44fa-8607-1f6b75fc7f46/volumes" Oct 30 22:15:18.102253 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:18.102212 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:15:19.181059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:19.181027 5036 scope.go:122] "RemoveContainer" containerID="9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952" Oct 30 22:15:19.181477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:19.181190 5036 scope.go:122] "RemoveContainer" containerID="9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92" Oct 30 22:15:19.181477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:19.181314 5036 scope.go:122] "RemoveContainer" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" Oct 30 22:15:20.185954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.185924 5036 scope.go:122] "RemoveContainer" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" Oct 30 22:15:20.187459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.187439 5036 scope.go:122] "RemoveContainer" containerID="09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978" Oct 30 22:15:20.188320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.188296 5036 scope.go:122] "RemoveContainer" containerID="21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91" Oct 30 22:15:20.188812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.188793 5036 scope.go:122] "RemoveContainer" containerID="79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0" Oct 30 22:15:20.191696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.190615 5036 scope.go:122] "RemoveContainer" containerID="c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1" Oct 30 22:15:20.191696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.190763 5036 scope.go:122] "RemoveContainer" containerID="7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9" Oct 30 22:15:20.486797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.486725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerStarted","Data":"ddbcfddd612ca93da05dcfa4e9c7d79839257834a7c988aa174da2410456820f"} Oct 30 22:15:20.488932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.488905 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerStarted","Data":"5df86fff3896170ec4d800e9031de708166ebf99d3349a79408f6e53ecfea854"} Oct 30 22:15:20.491043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.491021 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerStarted","Data":"4fc3a18191e2f5503ab6891a0f492ac7ee157ef5c610477ca9f8bf40c84c6777"} Oct 30 22:15:20.550553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.550495 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" podStartSLOduration=20.550475 podStartE2EDuration="20.550475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:20.522926864 +0000 UTC m=+2144.472881810" watchObservedRunningTime="2026-10-30 22:15:20.550475 +0000 UTC m=+2144.500429952" Oct 30 22:15:20.677038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:20.676889 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:15:21.181091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.180985 5036 scope.go:122] "RemoveContainer" containerID="008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb" Oct 30 22:15:21.181252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.181228 5036 scope.go:122] "RemoveContainer" containerID="61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e" Oct 30 22:15:21.181627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.181598 5036 scope.go:122] "RemoveContainer" containerID="b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40" Oct 30 22:15:21.181981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.181944 5036 scope.go:122] "RemoveContainer" containerID="3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee" Oct 30 22:15:21.504730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.504652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerStarted","Data":"0b7f0f46016aa18e7b8bb68739371c49f025d03b30b728056bc2366d6f76c91f"} Oct 30 22:15:21.506871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.506846 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerID="5df86fff3896170ec4d800e9031de708166ebf99d3349a79408f6e53ecfea854" exitCode=60 Oct 30 22:15:21.506992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.506921 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerDied","Data":"5df86fff3896170ec4d800e9031de708166ebf99d3349a79408f6e53ecfea854"} Oct 30 22:15:21.506992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.506973 5036 scope.go:122] "RemoveContainer" containerID="9359fc00c9a76b0d1f1286da1fe2c46ef1f2ddd1d9ecc94b022dffa2c1f45f92" Oct 30 22:15:21.507429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.507413 5036 scope.go:122] "RemoveContainer" containerID="5df86fff3896170ec4d800e9031de708166ebf99d3349a79408f6e53ecfea854" Oct 30 22:15:21.507861 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:21.507833 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf_odh-ai-gateway-infra(cd97a033-6160-4dc2-bde2-0ab637df4548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" Oct 30 22:15:21.509336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.509309 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerStarted","Data":"c912aa113e29781aee46532902694c41a2d76dbcb8c5c6089126f2c92f5afe5f"} Oct 30 22:15:21.515193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.512740 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerStarted","Data":"0d1a80f482849044a4821de534b42e2963b9a163db8a5991788560bf232e7f8d"} Oct 30 22:15:21.516893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.515969 5036 generic.go:356] "Generic (PLEG): container finished" podID="48f7a48b-c603-45e3-a20b-9630c280f391" containerID="4fc3a18191e2f5503ab6891a0f492ac7ee157ef5c610477ca9f8bf40c84c6777" exitCode=60 Oct 30 22:15:21.516893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.516141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerDied","Data":"4fc3a18191e2f5503ab6891a0f492ac7ee157ef5c610477ca9f8bf40c84c6777"} Oct 30 22:15:21.516893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.516510 5036 scope.go:122] "RemoveContainer" containerID="4fc3a18191e2f5503ab6891a0f492ac7ee157ef5c610477ca9f8bf40c84c6777" Oct 30 22:15:21.516893 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:21.516849 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6_odh-ai-gateway-infra(48f7a48b-c603-45e3-a20b-9630c280f391)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" Oct 30 22:15:21.525469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.525438 5036 generic.go:356] "Generic (PLEG): container finished" podID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerID="4696e886e012298e82dbde805ce49bf1d6176d13e227f064db9f7a75c7752d0c" exitCode=60 Oct 30 22:15:21.525606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.525565 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerDied","Data":"4696e886e012298e82dbde805ce49bf1d6176d13e227f064db9f7a75c7752d0c"} Oct 30 22:15:21.543042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.526022 5036 scope.go:122] "RemoveContainer" containerID="4696e886e012298e82dbde805ce49bf1d6176d13e227f064db9f7a75c7752d0c" Oct 30 22:15:21.543042 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:21.526608 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57_odh-ai-gateway-infra(daf049fb-3442-41b6-abe4-b43daeaf22e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" Oct 30 22:15:21.558871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.553286 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerStarted","Data":"fd4b14d4be6229c06ca765c5fdecdce35ea048754a0f5e0354d202f10396ed9b"} Oct 30 22:15:21.570497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.562656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerStarted","Data":"999d0126684905db023ab0b6f2fb24aed0bba093381cbe0503f889299e9d6c52"} Oct 30 22:15:21.581230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.581193 5036 generic.go:356] "Generic (PLEG): container finished" podID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerID="ddbcfddd612ca93da05dcfa4e9c7d79839257834a7c988aa174da2410456820f" exitCode=60 Oct 30 22:15:21.581526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.581500 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerDied","Data":"ddbcfddd612ca93da05dcfa4e9c7d79839257834a7c988aa174da2410456820f"} Oct 30 22:15:21.582296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.582234 5036 scope.go:122] "RemoveContainer" containerID="ddbcfddd612ca93da05dcfa4e9c7d79839257834a7c988aa174da2410456820f" Oct 30 22:15:21.582973 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:21.582936 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz_odh-ai-gateway-infra(93159d6d-45b9-4b2c-8bcd-4dd2d3507b50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" Oct 30 22:15:21.654355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.648435 5036 scope.go:122] "RemoveContainer" containerID="33cadd5c35bf9268fb1131e0ee8a79564f5eec0ac0b22ed90d82d6c6fa38764e" Oct 30 22:15:21.654785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.654632 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:21.673485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.673459 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:21.689791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.689771 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:21.731775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.731722 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" podStartSLOduration=21.731709382 podStartE2EDuration="21.731709382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.702552427 +0000 UTC m=+2145.652507373" watchObservedRunningTime="2026-10-30 22:15:21.731709382 +0000 UTC m=+2145.681664327" Oct 30 22:15:21.853684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.853657 5036 scope.go:122] "RemoveContainer" containerID="1b5085e216500455ecd974910485eaeb2ff5ba8e35c75a209227c147e143d057" Oct 30 22:15:21.869708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:21.869683 5036 scope.go:122] "RemoveContainer" containerID="9d00b001e01edfa10cd7025d2579cc34de4c85a2ec95c379bf94f80b464a4952" Oct 30 22:15:22.180787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.180758 5036 scope.go:122] "RemoveContainer" containerID="2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d" Oct 30 22:15:22.181364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.181337 5036 scope.go:122] "RemoveContainer" containerID="2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd" Oct 30 22:15:22.560232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.560192 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:22.599111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.599083 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerStarted","Data":"7b724d0430bf4ac3a987404bdc88a3522d10d68fc0880f771321cf863cf81ac3"} Oct 30 22:15:22.601627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.601587 5036 generic.go:356] "Generic (PLEG): container finished" podID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerID="fd4b14d4be6229c06ca765c5fdecdce35ea048754a0f5e0354d202f10396ed9b" exitCode=60 Oct 30 22:15:22.601741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.601652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerDied","Data":"fd4b14d4be6229c06ca765c5fdecdce35ea048754a0f5e0354d202f10396ed9b"} Oct 30 22:15:22.601741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.601719 5036 scope.go:122] "RemoveContainer" containerID="09e13c982b00f707f786c9a0bedd4ca8ce0ba4184a8ba5bcbd12b63235d49978" Oct 30 22:15:22.602127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.602107 5036 scope.go:122] "RemoveContainer" containerID="fd4b14d4be6229c06ca765c5fdecdce35ea048754a0f5e0354d202f10396ed9b" Oct 30 22:15:22.602546 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:22.602520 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b_odh-ai-gateway-infra(baf9a179-a462-4c72-aaf5-cd1af5717093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" Oct 30 22:15:22.606447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.606419 5036 generic.go:356] "Generic (PLEG): container finished" podID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerID="999d0126684905db023ab0b6f2fb24aed0bba093381cbe0503f889299e9d6c52" exitCode=60 Oct 30 22:15:22.606662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.606473 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerDied","Data":"999d0126684905db023ab0b6f2fb24aed0bba093381cbe0503f889299e9d6c52"} Oct 30 22:15:22.607248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.607121 5036 scope.go:122] "RemoveContainer" containerID="999d0126684905db023ab0b6f2fb24aed0bba093381cbe0503f889299e9d6c52" Oct 30 22:15:22.607700 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:22.607672 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s_odh-ai-gateway-infra(8527b219-c5dc-45ae-b088-0db98e8e1042)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" Oct 30 22:15:22.628560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.627528 5036 scope.go:122] "RemoveContainer" containerID="c86afed4cf44f1e8d4520ba66142eb9dba5c12798c19e7a47351d46e9eeccfc1" Oct 30 22:15:22.628560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.627659 5036 generic.go:356] "Generic (PLEG): container finished" podID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerID="0b7f0f46016aa18e7b8bb68739371c49f025d03b30b728056bc2366d6f76c91f" exitCode=60 Oct 30 22:15:22.628560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.627850 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerDied","Data":"0b7f0f46016aa18e7b8bb68739371c49f025d03b30b728056bc2366d6f76c91f"} Oct 30 22:15:22.642170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.641475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerStarted","Data":"c073c3509c915cf2c6360e6697605495a86969392caf8e1a460bd8ac4296d83c"} Oct 30 22:15:22.650067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.650032 5036 generic.go:356] "Generic (PLEG): container finished" podID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerID="1fa850917e5c4ce015dfe0f67d5e803ef58f4febed4f8efdd395a0bc6ba8136f" exitCode=60 Oct 30 22:15:22.650205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.650138 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerDied","Data":"1fa850917e5c4ce015dfe0f67d5e803ef58f4febed4f8efdd395a0bc6ba8136f"} Oct 30 22:15:22.653696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.653665 5036 scope.go:122] "RemoveContainer" containerID="1fa850917e5c4ce015dfe0f67d5e803ef58f4febed4f8efdd395a0bc6ba8136f" Oct 30 22:15:22.654249 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:22.654218 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq_odh-ai-gateway-infra(e48a647e-1b9e-4248-aa22-1349c838eb04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" Oct 30 22:15:22.656956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.655820 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerID="c912aa113e29781aee46532902694c41a2d76dbcb8c5c6089126f2c92f5afe5f" exitCode=60 Oct 30 22:15:22.656956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.655934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerDied","Data":"c912aa113e29781aee46532902694c41a2d76dbcb8c5c6089126f2c92f5afe5f"} Oct 30 22:15:22.656956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.656437 5036 scope.go:122] "RemoveContainer" containerID="c912aa113e29781aee46532902694c41a2d76dbcb8c5c6089126f2c92f5afe5f" Oct 30 22:15:22.656956 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:22.656883 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz_odh-ai-gateway-infra(8c761ec4-d3b0-4663-88e7-2de45be4f9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" Oct 30 22:15:22.664912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.664873 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" podStartSLOduration=22.664858315 podStartE2EDuration="22.664858315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:22.649590919 +0000 UTC m=+2146.599545867" watchObservedRunningTime="2026-10-30 22:15:22.664858315 +0000 UTC m=+2146.614813261" Oct 30 22:15:22.670760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.667998 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:22.670760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.668176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerStarted","Data":"f205bd8741442a2edbdd9214780c7f1e270deb90731ce198fb4e7cb8e67deb41"} Oct 30 22:15:22.673650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.673629 5036 generic.go:356] "Generic (PLEG): container finished" podID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerID="0d1a80f482849044a4821de534b42e2963b9a163db8a5991788560bf232e7f8d" exitCode=60 Oct 30 22:15:22.673764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.673679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerDied","Data":"0d1a80f482849044a4821de534b42e2963b9a163db8a5991788560bf232e7f8d"} Oct 30 22:15:22.674215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.674201 5036 scope.go:122] "RemoveContainer" containerID="0d1a80f482849044a4821de534b42e2963b9a163db8a5991788560bf232e7f8d" Oct 30 22:15:22.674758 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:22.674704 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv_odh-ai-gateway-infra(e6802e98-f7d3-4b61-9d03-ddec2b4442fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" Oct 30 22:15:22.689257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.689228 5036 scope.go:122] "RemoveContainer" containerID="21108028849a4e0dfeb39187d2a522eca2057b95bc8fc3efca79d127a9b36f91" Oct 30 22:15:22.701417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.701383 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:22.744903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.744877 5036 scope.go:122] "RemoveContainer" containerID="61613a6fca832a27e1f0c6238da8a1e15a1a371e5bcbd621557f50ce77fb424e" Oct 30 22:15:22.749451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.749422 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:22.775334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.775214 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:22.796006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.795980 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:22.948163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.948005 5036 scope.go:122] "RemoveContainer" containerID="7b0f67ed2761b304227f82546e50517464878288c2d413b723583eb85575c6f9" Oct 30 22:15:22.975852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:22.975376 5036 scope.go:122] "RemoveContainer" containerID="79ee809a51238bc649286218cfdc427e15fef02cdfb850237c372bcdc7dde6a0" Oct 30 22:15:23.014607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.014112 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:23.066185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.065379 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\") pod \"48f7a48b-c603-45e3-a20b-9630c280f391\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " Oct 30 22:15:23.066185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.065676 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\" (UniqueName: \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\") pod \"48f7a48b-c603-45e3-a20b-9630c280f391\" (UID: \"48f7a48b-c603-45e3-a20b-9630c280f391\") " Oct 30 22:15:23.066185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.065995 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca" pod "48f7a48b-c603-45e3-a20b-9630c280f391" (UID: "48f7a48b-c603-45e3-a20b-9630c280f391"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.069983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.069949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks" pod "48f7a48b-c603-45e3-a20b-9630c280f391" (UID: "48f7a48b-c603-45e3-a20b-9630c280f391"). InnerVolumeSpecName "kube-api-access-wnmks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.070703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.070450 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f7a48b-c603-45e3-a20b-9630c280f391-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.070703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.070479 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmks\" (UniqueName: \"kubernetes.io/projected/48f7a48b-c603-45e3-a20b-9630c280f391-kube-api-access-wnmks\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.101186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.101151 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:23.111785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.111761 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:23.118097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.118074 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:23.157771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.157749 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\") pod \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415367 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\") pod \"cd97a033-6160-4dc2-bde2-0ab637df4548\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415563 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\" (UniqueName: \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\") pod \"b6bd0e01-0001-426e-b7a3-e8140661136f\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415608 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\") pod \"daf049fb-3442-41b6-abe4-b43daeaf22e2\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415657 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\" (UniqueName: \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\") pod \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\" (UID: \"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\" (UniqueName: \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\") pod \"daf049fb-3442-41b6-abe4-b43daeaf22e2\" (UID: \"daf049fb-3442-41b6-abe4-b43daeaf22e2\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415836 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\" (UniqueName: \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\") pod \"cd97a033-6160-4dc2-bde2-0ab637df4548\" (UID: \"cd97a033-6160-4dc2-bde2-0ab637df4548\") " Oct 30 22:15:23.423749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.415888 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\") pod \"b6bd0e01-0001-426e-b7a3-e8140661136f\" (UID: \"b6bd0e01-0001-426e-b7a3-e8140661136f\") " Oct 30 22:15:23.426344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.426307 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca" pod "b6bd0e01-0001-426e-b7a3-e8140661136f" (UID: "b6bd0e01-0001-426e-b7a3-e8140661136f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.430882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.430651 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8" pod "daf049fb-3442-41b6-abe4-b43daeaf22e2" (UID: "daf049fb-3442-41b6-abe4-b43daeaf22e2"). InnerVolumeSpecName "kube-api-access-ld5k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.431697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.431191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca" pod "cd97a033-6160-4dc2-bde2-0ab637df4548" (UID: "cd97a033-6160-4dc2-bde2-0ab637df4548"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.432958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.432236 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g" pod "cd97a033-6160-4dc2-bde2-0ab637df4548" (UID: "cd97a033-6160-4dc2-bde2-0ab637df4548"). InnerVolumeSpecName "kube-api-access-l5v6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.433478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.433192 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca" pod "daf049fb-3442-41b6-abe4-b43daeaf22e2" (UID: "daf049fb-3442-41b6-abe4-b43daeaf22e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.436162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.436067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94" pod "b6bd0e01-0001-426e-b7a3-e8140661136f" (UID: "b6bd0e01-0001-426e-b7a3-e8140661136f"). InnerVolumeSpecName "kube-api-access-znp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.503923 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca" pod "93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" (UID: "93159d6d-45b9-4b2c-8bcd-4dd2d3507b50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542774 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5k8\" (UniqueName: \"kubernetes.io/projected/daf049fb-3442-41b6-abe4-b43daeaf22e2-kube-api-access-ld5k8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542803 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5v6g\" (UniqueName: \"kubernetes.io/projected/cd97a033-6160-4dc2-bde2-0ab637df4548-kube-api-access-l5v6g\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542822 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bd0e01-0001-426e-b7a3-e8140661136f-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542837 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542851 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd97a033-6160-4dc2-bde2-0ab637df4548-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542865 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znp94\" (UniqueName: \"kubernetes.io/projected/b6bd0e01-0001-426e-b7a3-e8140661136f-kube-api-access-znp94\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.542879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf049fb-3442-41b6-abe4-b43daeaf22e2-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.561292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.543433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5" pod "93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" (UID: "93159d6d-45b9-4b2c-8bcd-4dd2d3507b50"). InnerVolumeSpecName "kube-api-access-t5sr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.661540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.661501 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:23.666197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.666150 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5sr5\" (UniqueName: \"kubernetes.io/projected/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50-kube-api-access-t5sr5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.696385 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" event={"ID":"93159d6d-45b9-4b2c-8bcd-4dd2d3507b50","Type":"ContainerDied","Data":"c2b5a829408a320768c31524e01dba2bbab8cfa442cd59b8d027e3a489740f66"} Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.696404 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz" Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.696466 5036 scope.go:122] "RemoveContainer" containerID="ddbcfddd612ca93da05dcfa4e9c7d79839257834a7c988aa174da2410456820f" Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.699518 5036 generic.go:356] "Generic (PLEG): container finished" podID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerID="5d17da30f749de5832096851c80a2a941462711c527d6e2db4a683b953964e73" exitCode=60 Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.699575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerDied","Data":"5d17da30f749de5832096851c80a2a941462711c527d6e2db4a683b953964e73"} Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.700091 5036 scope.go:122] "RemoveContainer" containerID="5d17da30f749de5832096851c80a2a941462711c527d6e2db4a683b953964e73" Oct 30 22:15:23.701318 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:23.700540 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f_odh-ai-gateway-infra(9ca06018-5e9c-4cc9-b360-9ead31686bf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" Oct 30 22:15:23.735922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.734640 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" event={"ID":"b6bd0e01-0001-426e-b7a3-e8140661136f","Type":"ContainerDied","Data":"6d8abf7f9fecc54f493b52bb038179aa68ecd4eafde98a9fafb9e5afbe895019"} Oct 30 22:15:23.735922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.734798 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw" Oct 30 22:15:23.746323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.746252 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" Oct 30 22:15:23.751899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.747352 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf" event={"ID":"cd97a033-6160-4dc2-bde2-0ab637df4548","Type":"ContainerDied","Data":"2447be663b64590b840fb7220572c1c90d1faa9b76032e4d5b3ad8f76f24b6a2"} Oct 30 22:15:23.752201 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.752175 5036 generic.go:356] "Generic (PLEG): container finished" podID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerID="c073c3509c915cf2c6360e6697605495a86969392caf8e1a460bd8ac4296d83c" exitCode=60 Oct 30 22:15:23.752385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.752364 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerDied","Data":"c073c3509c915cf2c6360e6697605495a86969392caf8e1a460bd8ac4296d83c"} Oct 30 22:15:23.753010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.752988 5036 scope.go:122] "RemoveContainer" containerID="c073c3509c915cf2c6360e6697605495a86969392caf8e1a460bd8ac4296d83c" Oct 30 22:15:23.754719 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:23.754687 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-lqx56_odh-ai-gateway-infra(13459fce-bfbc-45e0-8e1a-7c6b567c76e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" Oct 30 22:15:23.766452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.766425 5036 scope.go:122] "RemoveContainer" containerID="2a02da13d3cad71b8c3a879244e0fede5431bbd3f25ebe674fc5af95466e766d" Oct 30 22:15:23.797499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.797467 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:23.812891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.810841 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:23.823594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.823166 5036 generic.go:356] "Generic (PLEG): container finished" podID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerID="f205bd8741442a2edbdd9214780c7f1e270deb90731ce198fb4e7cb8e67deb41" exitCode=60 Oct 30 22:15:23.823594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.823296 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerDied","Data":"f205bd8741442a2edbdd9214780c7f1e270deb90731ce198fb4e7cb8e67deb41"} Oct 30 22:15:23.826776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.825885 5036 scope.go:122] "RemoveContainer" containerID="f205bd8741442a2edbdd9214780c7f1e270deb90731ce198fb4e7cb8e67deb41" Oct 30 22:15:23.828070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.827650 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889975-xpmxz"] Oct 30 22:15:23.828605 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:23.828345 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2_odh-ai-gateway-infra(2e4148eb-52e6-4c5b-ba64-3948bc5533ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" Oct 30 22:15:23.857548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.857523 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:23.892180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.891983 5036 scope.go:122] "RemoveContainer" containerID="0b7f0f46016aa18e7b8bb68739371c49f025d03b30b728056bc2366d6f76c91f" Oct 30 22:15:23.900099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.900066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" event={"ID":"48f7a48b-c603-45e3-a20b-9630c280f391","Type":"ContainerDied","Data":"8633d0fa86b0460bcc81b26c31ceb191607677e2fe4af78e0a12267b3a125999"} Oct 30 22:15:23.900224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.900201 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6" Oct 30 22:15:23.924744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.924250 5036 scope.go:122] "RemoveContainer" containerID="5df86fff3896170ec4d800e9031de708166ebf99d3349a79408f6e53ecfea854" Oct 30 22:15:23.928174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.925463 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" event={"ID":"daf049fb-3442-41b6-abe4-b43daeaf22e2","Type":"ContainerDied","Data":"8b207bac9199e8f36bbfb67cd94b6ec553039640cfce777c5a55f6dde974542a"} Oct 30 22:15:23.928174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.925571 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57" Oct 30 22:15:23.931877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.931658 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerID="7b724d0430bf4ac3a987404bdc88a3522d10d68fc0880f771321cf863cf81ac3" exitCode=60 Oct 30 22:15:23.931877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.931730 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerDied","Data":"7b724d0430bf4ac3a987404bdc88a3522d10d68fc0880f771321cf863cf81ac3"} Oct 30 22:15:23.945696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.945628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerStarted","Data":"1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c"} Oct 30 22:15:23.946471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:23.946449 5036 scope.go:122] "RemoveContainer" containerID="1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c" Oct 30 22:15:23.947126 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:23.947050 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj_odh-ai-gateway-infra(7dd08765-1f01-4991-beef-acae8a938a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" podUID="7dd08765-1f01-4991-beef-acae8a938a12" Oct 30 22:15:24.136295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.136178 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:24.362707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.362659 5036 scope.go:122] "RemoveContainer" containerID="3fa1e921330f68db1d12a83ddaf93d28b2daba1fd09fb67f92cdc911ed9c2bee" Oct 30 22:15:24.383614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.383576 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" path="/var/lib/kubelet/pods/93159d6d-45b9-4b2c-8bcd-4dd2d3507b50/volumes" Oct 30 22:15:24.428871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.428846 5036 scope.go:122] "RemoveContainer" containerID="b5b0c2d8a8d6753f432260f66f984215e5dcf6662c15d81df5f7e767cd590f40" Oct 30 22:15:24.438405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.438380 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889975-542rw"] Oct 30 22:15:24.444131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.444107 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:24.456096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.456069 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:24.468802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.463532 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:24.481391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.481320 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889975-kp2cf"] Oct 30 22:15:24.499052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.498881 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\" (UniqueName: \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\") pod \"8527b219-c5dc-45ae-b088-0db98e8e1042\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " Oct 30 22:15:24.499162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.499129 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\") pod \"8527b219-c5dc-45ae-b088-0db98e8e1042\" (UID: \"8527b219-c5dc-45ae-b088-0db98e8e1042\") " Oct 30 22:15:24.502915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.500177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca" pod "8527b219-c5dc-45ae-b088-0db98e8e1042" (UID: "8527b219-c5dc-45ae-b088-0db98e8e1042"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.509780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.509665 5036 scope.go:122] "RemoveContainer" containerID="4fc3a18191e2f5503ab6891a0f492ac7ee157ef5c610477ca9f8bf40c84c6777" Oct 30 22:15:24.570609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.570527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv" pod "8527b219-c5dc-45ae-b088-0db98e8e1042" (UID: "8527b219-c5dc-45ae-b088-0db98e8e1042"). InnerVolumeSpecName "kube-api-access-b9vwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.592291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.592187 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:24.595069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.595044 5036 scope.go:122] "RemoveContainer" containerID="4696e886e012298e82dbde805ce49bf1d6176d13e227f064db9f7a75c7752d0c" Oct 30 22:15:24.600596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.600560 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\") pod \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " Oct 30 22:15:24.600721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.600661 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\" (UniqueName: \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\") pod \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\" (UID: \"8c761ec4-d3b0-4663-88e7-2de45be4f9f7\") " Oct 30 22:15:24.601047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.601026 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca" pod "8c761ec4-d3b0-4663-88e7-2de45be4f9f7" (UID: "8c761ec4-d3b0-4663-88e7-2de45be4f9f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.603396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.602422 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vwv\" (UniqueName: \"kubernetes.io/projected/8527b219-c5dc-45ae-b088-0db98e8e1042-kube-api-access-b9vwv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.603396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.602449 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.603396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.602463 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8527b219-c5dc-45ae-b088-0db98e8e1042-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.604770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.604751 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:24.604916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.604783 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm" pod "8c761ec4-d3b0-4663-88e7-2de45be4f9f7" (UID: "8c761ec4-d3b0-4663-88e7-2de45be4f9f7"). InnerVolumeSpecName "kube-api-access-drbrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.613557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.613527 5036 scope.go:122] "RemoveContainer" containerID="008d544ed61cf868d01bcd25aba9e461f93e796fb361afd50f5a6de034eaefbb" Oct 30 22:15:24.615153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.613753 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5313f0-29889975-59gv6"] Oct 30 22:15:24.634741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.624488 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:24.634741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.631283 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889975-nkk57"] Oct 30 22:15:24.682882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.682582 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:24.703571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.703535 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\" (UniqueName: \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\") pod \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " Oct 30 22:15:24.703927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.703905 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\") pod \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\" (UID: \"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e\") " Oct 30 22:15:24.705121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.705095 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbrm\" (UniqueName: \"kubernetes.io/projected/8c761ec4-d3b0-4663-88e7-2de45be4f9f7-kube-api-access-drbrm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.705577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.705555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca" pod "7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" (UID: "7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.719956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.719904 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn" pod "7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" (UID: "7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e"). InnerVolumeSpecName "kube-api-access-jf4jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.803085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.803056 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:24.803797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.803515 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:24.807258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.807213 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4jn\" (UniqueName: \"kubernetes.io/projected/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-kube-api-access-jf4jn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.807258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.807246 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.813730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.813710 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:24.819383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:24.819359 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:25.000681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.000644 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\" (UniqueName: \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\") pod \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " Oct 30 22:15:25.000885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.000734 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\") pod \"e48a647e-1b9e-4248-aa22-1349c838eb04\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " Oct 30 22:15:25.000885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.000799 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\") pod \"baf9a179-a462-4c72-aaf5-cd1af5717093\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " Oct 30 22:15:25.001010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.000911 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\") pod \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\" (UID: \"e6802e98-f7d3-4b61-9d03-ddec2b4442fb\") " Oct 30 22:15:25.001010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.000982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\" (UniqueName: \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\") pod \"baf9a179-a462-4c72-aaf5-cd1af5717093\" (UID: \"baf9a179-a462-4c72-aaf5-cd1af5717093\") " Oct 30 22:15:25.001118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.001031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\" (UniqueName: \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\") pod \"e48a647e-1b9e-4248-aa22-1349c838eb04\" (UID: \"e48a647e-1b9e-4248-aa22-1349c838eb04\") " Oct 30 22:15:25.003629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.003597 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca" pod "e6802e98-f7d3-4b61-9d03-ddec2b4442fb" (UID: "e6802e98-f7d3-4b61-9d03-ddec2b4442fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.004318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.004289 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca" pod "e48a647e-1b9e-4248-aa22-1349c838eb04" (UID: "e48a647e-1b9e-4248-aa22-1349c838eb04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.006019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.005986 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7" pod "e6802e98-f7d3-4b61-9d03-ddec2b4442fb" (UID: "e6802e98-f7d3-4b61-9d03-ddec2b4442fb"). InnerVolumeSpecName "kube-api-access-pwfl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.006333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.006309 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca" pod "baf9a179-a462-4c72-aaf5-cd1af5717093" (UID: "baf9a179-a462-4c72-aaf5-cd1af5717093"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.008796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.008772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk" pod "baf9a179-a462-4c72-aaf5-cd1af5717093" (UID: "baf9a179-a462-4c72-aaf5-cd1af5717093"). InnerVolumeSpecName "kube-api-access-prslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.013528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.013500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh" pod "e48a647e-1b9e-4248-aa22-1349c838eb04" (UID: "e48a647e-1b9e-4248-aa22-1349c838eb04"). InnerVolumeSpecName "kube-api-access-tnxkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.016939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.016909 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" event={"ID":"7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e","Type":"ContainerDied","Data":"281dd6d624e68501dc7f1eee513db2ab555bc1553cbb63f51ae469815575a2d4"} Oct 30 22:15:25.017048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.016949 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh" Oct 30 22:15:25.017048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.017005 5036 scope.go:122] "RemoveContainer" containerID="7b724d0430bf4ac3a987404bdc88a3522d10d68fc0880f771321cf863cf81ac3" Oct 30 22:15:25.019287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.019253 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" Oct 30 22:15:25.019550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.019341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b" event={"ID":"baf9a179-a462-4c72-aaf5-cd1af5717093","Type":"ContainerDied","Data":"8894c35f06436e018397c7f76f7ec1470329244af88dc80cfcefe36d4093b8ac"} Oct 30 22:15:25.021675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.021647 5036 generic.go:356] "Generic (PLEG): container finished" podID="7dd08765-1f01-4991-beef-acae8a938a12" containerID="1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c" exitCode=60 Oct 30 22:15:25.021769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.021736 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerDied","Data":"1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c"} Oct 30 22:15:25.022278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.022244 5036 scope.go:122] "RemoveContainer" containerID="1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c" Oct 30 22:15:25.022630 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:25.022611 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj_odh-ai-gateway-infra(7dd08765-1f01-4991-beef-acae8a938a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" podUID="7dd08765-1f01-4991-beef-acae8a938a12" Oct 30 22:15:25.024998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.024977 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" event={"ID":"8527b219-c5dc-45ae-b088-0db98e8e1042","Type":"ContainerDied","Data":"abe1e6c826de10c06a98579f00c35cf679fd4eb85585d809181d5403a348192d"} Oct 30 22:15:25.024998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.024994 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s" Oct 30 22:15:25.034978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.034867 5036 scope.go:122] "RemoveContainer" containerID="fd4b14d4be6229c06ca765c5fdecdce35ea048754a0f5e0354d202f10396ed9b" Oct 30 22:15:25.047503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.046329 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" event={"ID":"e48a647e-1b9e-4248-aa22-1349c838eb04","Type":"ContainerDied","Data":"98c035d8ffc56f7a20b90c049f887681a2cc9d211cdf48aefab006a214809051"} Oct 30 22:15:25.047503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.046417 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq" Oct 30 22:15:25.052196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.052167 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" event={"ID":"8c761ec4-d3b0-4663-88e7-2de45be4f9f7","Type":"ContainerDied","Data":"7a8ce4a7faf5409c106bf9e6777d5541005fd82b36d92975bc7c66f1b28b98c4"} Oct 30 22:15:25.053837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.052283 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz" Oct 30 22:15:25.058790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.058769 5036 scope.go:122] "RemoveContainer" containerID="2f3b84d171220545b5796ff4c9db952b09adbc650a64948cf0952b77e3292dbd" Oct 30 22:15:25.075213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.075183 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" Oct 30 22:15:25.075862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.075831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv" event={"ID":"e6802e98-f7d3-4b61-9d03-ddec2b4442fb","Type":"ContainerDied","Data":"6318f8b0f5734e4988acd01b0f78a15179d0cc1501aff082fb7441e4fc0e2d63"} Oct 30 22:15:25.104925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104891 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prslk\" (UniqueName: \"kubernetes.io/projected/baf9a179-a462-4c72-aaf5-cd1af5717093-kube-api-access-prslk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.104925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104925 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnxkh\" (UniqueName: \"kubernetes.io/projected/e48a647e-1b9e-4248-aa22-1349c838eb04-kube-api-access-tnxkh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.105182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104940 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfl7\" (UniqueName: \"kubernetes.io/projected/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-kube-api-access-pwfl7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.105182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104954 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48a647e-1b9e-4248-aa22-1349c838eb04-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.105182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104979 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf9a179-a462-4c72-aaf5-cd1af5717093-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.105182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.104992 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6802e98-f7d3-4b61-9d03-ddec2b4442fb-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.116530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.116410 5036 scope.go:122] "RemoveContainer" containerID="999d0126684905db023ab0b6f2fb24aed0bba093381cbe0503f889299e9d6c52" Oct 30 22:15:25.121496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.120019 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:25.128516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.128493 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889975-68sbh"] Oct 30 22:15:25.204370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.203103 5036 scope.go:122] "RemoveContainer" containerID="1fa850917e5c4ce015dfe0f67d5e803ef58f4febed4f8efdd395a0bc6ba8136f" Oct 30 22:15:25.233087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.232969 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:25.238293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.238192 5036 scope.go:122] "RemoveContainer" containerID="c912aa113e29781aee46532902694c41a2d76dbcb8c5c6089126f2c92f5afe5f" Oct 30 22:15:25.240106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.240050 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889975-7269b"] Oct 30 22:15:25.248597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.248574 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:25.256007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.255954 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889975-4sbvq"] Oct 30 22:15:25.263781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.263726 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:25.271467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.271437 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889975-gx2tz"] Oct 30 22:15:25.281161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.281139 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:25.289048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.288914 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889975-n2gkv"] Oct 30 22:15:25.309149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.309076 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:25.325910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.325880 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889975-qq86s"] Oct 30 22:15:25.354670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.354639 5036 scope.go:122] "RemoveContainer" containerID="0d1a80f482849044a4821de534b42e2963b9a163db8a5991788560bf232e7f8d" Oct 30 22:15:25.370543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.370520 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:25.413681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.412181 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\") pod \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " Oct 30 22:15:25.413681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.412479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\" (UniqueName: \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\") pod \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\" (UID: \"9ca06018-5e9c-4cc9-b360-9ead31686bf1\") " Oct 30 22:15:25.414460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.414436 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca" pod "9ca06018-5e9c-4cc9-b360-9ead31686bf1" (UID: "9ca06018-5e9c-4cc9-b360-9ead31686bf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.419606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.419579 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884" pod "9ca06018-5e9c-4cc9-b360-9ead31686bf1" (UID: "9ca06018-5e9c-4cc9-b360-9ead31686bf1"). InnerVolumeSpecName "kube-api-access-fv884". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.467153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.467126 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:25.469335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.469313 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:25.514500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.514473 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\" (UniqueName: \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\") pod \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " Oct 30 22:15:25.514688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.514674 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\") pod \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\" (UID: \"2e4148eb-52e6-4c5b-ba64-3948bc5533ed\") " Oct 30 22:15:25.514750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.514706 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\" (UniqueName: \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\") pod \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " Oct 30 22:15:25.514811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.514795 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\") pod \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\" (UID: \"13459fce-bfbc-45e0-8e1a-7c6b567c76e9\") " Oct 30 22:15:25.515006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.514985 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca" pod "2e4148eb-52e6-4c5b-ba64-3948bc5533ed" (UID: "2e4148eb-52e6-4c5b-ba64-3948bc5533ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.515318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.515252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca" pod "13459fce-bfbc-45e0-8e1a-7c6b567c76e9" (UID: "13459fce-bfbc-45e0-8e1a-7c6b567c76e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.515650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.515634 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca06018-5e9c-4cc9-b360-9ead31686bf1-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.515744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.515652 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.515744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.515662 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.515744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.515671 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv884\" (UniqueName: \"kubernetes.io/projected/9ca06018-5e9c-4cc9-b360-9ead31686bf1-kube-api-access-fv884\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.517147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.517118 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj" pod "13459fce-bfbc-45e0-8e1a-7c6b567c76e9" (UID: "13459fce-bfbc-45e0-8e1a-7c6b567c76e9"). InnerVolumeSpecName "kube-api-access-xqwhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.517638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.517618 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5" pod "2e4148eb-52e6-4c5b-ba64-3948bc5533ed" (UID: "2e4148eb-52e6-4c5b-ba64-3948bc5533ed"). InnerVolumeSpecName "kube-api-access-fcps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.617122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.617044 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwhj\" (UniqueName: \"kubernetes.io/projected/13459fce-bfbc-45e0-8e1a-7c6b567c76e9-kube-api-access-xqwhj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.617122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.617077 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcps5\" (UniqueName: \"kubernetes.io/projected/2e4148eb-52e6-4c5b-ba64-3948bc5533ed-kube-api-access-fcps5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.622049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.622016 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:25.678099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.678060 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" podUID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" containerName="payload-processing" containerID="cri-o://e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825" gracePeriod=25 Oct 30 22:15:25.985146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:25.985109 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:15:26.093310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.093227 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" Oct 30 22:15:26.093310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.093228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f" event={"ID":"9ca06018-5e9c-4cc9-b360-9ead31686bf1","Type":"ContainerDied","Data":"8210e2e19963a32e67762d5d4bd9725870993489a66e2ce7dbdcaf83870e7071"} Oct 30 22:15:26.093570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.093358 5036 scope.go:122] "RemoveContainer" containerID="5d17da30f749de5832096851c80a2a941462711c527d6e2db4a683b953964e73" Oct 30 22:15:26.094993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.094967 5036 generic.go:356] "Generic (PLEG): container finished" podID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" containerID="e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825" exitCode=0 Oct 30 22:15:26.095128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.095094 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" Oct 30 22:15:26.095206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.095124 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" event={"ID":"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df","Type":"ContainerDied","Data":"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825"} Oct 30 22:15:26.095206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.095166 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr" event={"ID":"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df","Type":"ContainerDied","Data":"b28168651eb618dd37504028b7329c525e1633e435a386ad546e54d237863fe6"} Oct 30 22:15:26.098406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.098384 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" Oct 30 22:15:26.098406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.098397 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56" event={"ID":"13459fce-bfbc-45e0-8e1a-7c6b567c76e9","Type":"ContainerDied","Data":"612680b46f103ddcbd9b9676e383143ebfb7f7d789f7606a846a049cc050f6dc"} Oct 30 22:15:26.102629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.102599 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" event={"ID":"2e4148eb-52e6-4c5b-ba64-3948bc5533ed","Type":"ContainerDied","Data":"da3b99e9a9ee748ca775e49ab2d70e5165cb5b60e6d9cb1d12b13596c748060e"} Oct 30 22:15:26.102739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.102609 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2" Oct 30 22:15:26.109029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.108993 5036 scope.go:122] "RemoveContainer" containerID="e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825" Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.140474 5036 scope.go:122] "RemoveContainer" containerID="e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825" Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.141245 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\") pod \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.141436 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\") pod \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.141486 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\") pod \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.141518 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\") pod \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\" (UID: \"02ef6a64-a0a8-451f-81d7-69f8ddcbb3df\") " Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:26.142845 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825\": container with ID starting with e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825 not found: ID does not exist" containerID="e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825" Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.142883 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825"} err="failed to get container status \"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825\": rpc error: code = NotFound desc = could not find container \"e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825\": container with ID starting with e11f217bf767496774a528650bce4eb7041a32203a5b6b117363caf04012e825 not found: ID does not exist" Oct 30 22:15:26.143226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.142909 5036 scope.go:122] "RemoveContainer" containerID="c073c3509c915cf2c6360e6697605495a86969392caf8e1a460bd8ac4296d83c" Oct 30 22:15:26.153463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.143626 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume" pod "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" (UID: "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:26.153463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.147570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume" pod "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" (UID: "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:26.154206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.154170 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz" pod "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" (UID: "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df"). InnerVolumeSpecName "kube-api-access-8q2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:26.157538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.157512 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials" pod "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" (UID: "02ef6a64-a0a8-451f-81d7-69f8ddcbb3df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:26.165104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.165080 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:26.174813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.173812 5036 scope.go:122] "RemoveContainer" containerID="f205bd8741442a2edbdd9214780c7f1e270deb90731ce198fb4e7cb8e67deb41" Oct 30 22:15:26.176589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.176571 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889975-w965f"] Oct 30 22:15:26.244964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.244910 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.244964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.244947 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.245305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.244977 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.245305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.244992 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2bz\" (UniqueName: \"kubernetes.io/projected/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df-kube-api-access-8q2bz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.248076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.248034 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" path="/var/lib/kubelet/pods/48f7a48b-c603-45e3-a20b-9630c280f391/volumes" Oct 30 22:15:26.248716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.248692 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" path="/var/lib/kubelet/pods/7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e/volumes" Oct 30 22:15:26.251523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.251498 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" path="/var/lib/kubelet/pods/8527b219-c5dc-45ae-b088-0db98e8e1042/volumes" Oct 30 22:15:26.252231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.252210 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" path="/var/lib/kubelet/pods/8c761ec4-d3b0-4663-88e7-2de45be4f9f7/volumes" Oct 30 22:15:26.253377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.253358 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" path="/var/lib/kubelet/pods/9ca06018-5e9c-4cc9-b360-9ead31686bf1/volumes" Oct 30 22:15:26.254084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.254065 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" path="/var/lib/kubelet/pods/b6bd0e01-0001-426e-b7a3-e8140661136f/volumes" Oct 30 22:15:26.254784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.254765 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" path="/var/lib/kubelet/pods/baf9a179-a462-4c72-aaf5-cd1af5717093/volumes" Oct 30 22:15:26.255866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.255847 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" path="/var/lib/kubelet/pods/cd97a033-6160-4dc2-bde2-0ab637df4548/volumes" Oct 30 22:15:26.256563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.256545 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" path="/var/lib/kubelet/pods/daf049fb-3442-41b6-abe4-b43daeaf22e2/volumes" Oct 30 22:15:26.274002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.273974 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" path="/var/lib/kubelet/pods/e48a647e-1b9e-4248-aa22-1349c838eb04/volumes" Oct 30 22:15:26.274663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.274635 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" path="/var/lib/kubelet/pods/e6802e98-f7d3-4b61-9d03-ddec2b4442fb/volumes" Oct 30 22:15:26.275490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.275472 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:26.275573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.275513 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d8c835-29889975-4bmm2"] Oct 30 22:15:26.275573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.275540 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:26.275573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.275560 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-lqx56"] Oct 30 22:15:26.367155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.367129 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:26.420142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.419839 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:15:26.428689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.428660 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-85b5z2hxr"] Oct 30 22:15:26.447772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.447749 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\" (UniqueName: \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\") pod \"7dd08765-1f01-4991-beef-acae8a938a12\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " Oct 30 22:15:26.447919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.447874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\") pod \"7dd08765-1f01-4991-beef-acae8a938a12\" (UID: \"7dd08765-1f01-4991-beef-acae8a938a12\") " Oct 30 22:15:26.448284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.448244 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca" pod "7dd08765-1f01-4991-beef-acae8a938a12" (UID: "7dd08765-1f01-4991-beef-acae8a938a12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:26.448659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.448641 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd08765-1f01-4991-beef-acae8a938a12-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.450365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.450344 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn" pod "7dd08765-1f01-4991-beef-acae8a938a12" (UID: "7dd08765-1f01-4991-beef-acae8a938a12"). InnerVolumeSpecName "kube-api-access-4hbbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:26.550496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:26.550415 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hbbn\" (UniqueName: \"kubernetes.io/projected/7dd08765-1f01-4991-beef-acae8a938a12-kube-api-access-4hbbn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:27.129878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:27.129846 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" Oct 30 22:15:27.129878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:27.129857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj" event={"ID":"7dd08765-1f01-4991-beef-acae8a938a12","Type":"ContainerDied","Data":"322273b5993e787f26df3178a4bfbb4ed7cd5ebe253c06304c879a5cdf4b8280"} Oct 30 22:15:27.130598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:27.129919 5036 scope.go:122] "RemoveContainer" containerID="1c44bfdb1d61e982cfeffad95bdfdcc8c09337b5f76972c95f93ab7f477cef8c" Oct 30 22:15:27.163389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:27.163363 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:27.168529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:27.168506 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c8ecf-29889975-l9jgj"] Oct 30 22:15:28.191061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:28.191028 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" path="/var/lib/kubelet/pods/02ef6a64-a0a8-451f-81d7-69f8ddcbb3df/volumes" Oct 30 22:15:28.191682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:28.191663 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" path="/var/lib/kubelet/pods/13459fce-bfbc-45e0-8e1a-7c6b567c76e9/volumes" Oct 30 22:15:28.192124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:28.192108 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" path="/var/lib/kubelet/pods/2e4148eb-52e6-4c5b-ba64-3948bc5533ed/volumes" Oct 30 22:15:28.192909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:28.192891 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd08765-1f01-4991-beef-acae8a938a12" path="/var/lib/kubelet/pods/7dd08765-1f01-4991-beef-acae8a938a12/volumes" Oct 30 22:15:39.040916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:39.040870 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258"] Oct 30 22:15:39.041434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:39.040938 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083f264c-ps258"] Oct 30 22:15:40.191078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:40.191041 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1bcf3c-1792-44e6-a20d-784c343eaa03" path="/var/lib/kubelet/pods/3e1bcf3c-1792-44e6-a20d-784c343eaa03/volumes" Oct 30 22:15:45.848279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.848233 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:15:45.849148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849128 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.849148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849148 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849164 5036 cpu_manager.go:414] "containerMap: removing container" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849169 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849177 5036 cpu_manager.go:414] "containerMap: removing container" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849182 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849193 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849198 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849204 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849208 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849215 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849220 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849229 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849234 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849240 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849246 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849274 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849279 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849286 5036 cpu_manager.go:414] "containerMap: removing container" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849291 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849296 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849300 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849311 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849315 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849321 5036 cpu_manager.go:414] "containerMap: removing container" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849326 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849332 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849336 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849342 5036 cpu_manager.go:414] "containerMap: removing container" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849346 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849353 5036 cpu_manager.go:414] "containerMap: removing container" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.849338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849358 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849368 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849372 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849378 5036 cpu_manager.go:414] "containerMap: removing container" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849382 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849394 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849399 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849405 5036 cpu_manager.go:414] "containerMap: removing container" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849409 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849415 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849419 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849426 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849430 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849438 5036 cpu_manager.go:414] "containerMap: removing container" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849442 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849450 5036 cpu_manager.go:414] "containerMap: removing container" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849454 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849465 5036 cpu_manager.go:414] "containerMap: removing container" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849469 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849476 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849480 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849486 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849490 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849497 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849501 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849511 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849515 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849525 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849530 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849537 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849541 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849548 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849552 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849559 5036 cpu_manager.go:414] "containerMap: removing container" podUID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849563 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849575 5036 cpu_manager.go:414] "containerMap: removing container" podUID="76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" containerName="revoke-keys" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849579 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" containerName="revoke-keys" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849587 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849591 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849598 5036 cpu_manager.go:414] "containerMap: removing container" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849603 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849609 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849613 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849620 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849624 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849635 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849639 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849761 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849768 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849774 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849781 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849788 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849793 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849798 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849803 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849808 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849813 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849819 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849825 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849829 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849835 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f86e877-9b24-44fa-8607-1f6b75fc7f46" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849841 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849846 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ef6a64-a0a8-451f-81d7-69f8ddcbb3df" containerName="payload-processing" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849853 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" containerName="revoke-keys" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849859 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849865 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849869 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849875 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849881 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849888 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6802e98-f7d3-4b61-9d03-ddec2b4442fb" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849894 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849900 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849906 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849911 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849915 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849921 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849927 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849932 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849939 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849946 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849952 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849958 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849963 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849969 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:45.851605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.849975 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:45.856047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.855023 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:45.860637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.860603 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"e2e-cmp-legacy-2aef6d-provider-cred\"" Oct 30 22:15:45.861066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.861042 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"payload-processing-external-model-e2e--ac81303a-dockercfg-5gr2l\"" Oct 30 22:15:45.861685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.861665 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"routing-overlay\"" Oct 30 22:15:45.861902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.861883 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"kube-root-ca.crt\"" Oct 30 22:15:45.861996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.861936 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"openshift-service-ca.crt\"" Oct 30 22:15:45.863258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.863228 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a39158\"" Oct 30 22:15:45.869203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.869182 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:15:45.949967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.949931 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:45.950187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.950152 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:45.950369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.950352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:45.950453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:45.950377 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9dg\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.076287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.064779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.076287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.064987 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.076287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.065179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.076287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.065224 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9dg\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.076287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.074011 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.102335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.089723 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.102335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.097997 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.123916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.114943 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9dg\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.177305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.175649 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:46.513361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.497595 5036 scope.go:122] "RemoveContainer" containerID="922cf120d2d0b89d8601da525f0cd204916807c2d9e9c6c72afb9cc34931b7f3" Oct 30 22:15:46.536163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.536125 5036 scope.go:122] "RemoveContainer" containerID="432c0df8b51555ce467200a316fc48a6907316481e07c0f302e7b5a09c7d4006" Oct 30 22:15:46.557882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.557850 5036 scope.go:122] "RemoveContainer" containerID="7bcce5ddcc78215b8d4533a06ad65d13d638760aa5566f059c3c64baf80391df" Oct 30 22:15:46.784570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:46.784538 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:15:47.475205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:47.475168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" event={"ID":"18a23f99-39db-4334-8644-b5f0b9ad237c","Type":"ContainerStarted","Data":"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24"} Oct 30 22:15:47.475707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:47.475236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" event={"ID":"18a23f99-39db-4334-8644-b5f0b9ad237c","Type":"ContainerStarted","Data":"125b3ec64d01dcfb691025dac7a9a7237bf910041f62e633b13e7dc0d50925f6"} Oct 30 22:15:47.475707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:47.475407 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:47.498781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:47.498730 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" podStartSLOduration=2.49871617 podStartE2EDuration="2.49871617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:47.493498375 +0000 UTC m=+2171.443453322" watchObservedRunningTime="2026-10-30 22:15:47.49871617 +0000 UTC m=+2171.448671120" Oct 30 22:15:48.101750 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:15:48.101702 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:15:49.067293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:49.066517 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2"] Oct 30 22:15:49.082180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:49.079975 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1697d7ca-vktk2"] Oct 30 22:15:50.190689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:50.190658 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3647372-3e68-429b-ad31-8bad33e78795" path="/var/lib/kubelet/pods/a3647372-3e68-429b-ad31-8bad33e78795/volumes" Oct 30 22:15:55.056566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.056530 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr"] Oct 30 22:15:55.057428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057411 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:55.057428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057428 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057442 5036 cpu_manager.go:414] "containerMap: removing container" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057449 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057461 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057466 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd08765-1f01-4991-beef-acae8a938a12" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057476 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057480 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa20ee-bcfc-460f-bdc2-2a8f34d2176e" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057491 5036 cpu_manager.go:414] "containerMap: removing container" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057495 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057509 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:55.057525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057514 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057536 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057541 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057557 5036 cpu_manager.go:414] "containerMap: removing container" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057562 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f7a48b-c603-45e3-a20b-9630c280f391" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057572 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057576 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8527b219-c5dc-45ae-b088-0db98e8e1042" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057696 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="13459fce-bfbc-45e0-8e1a-7c6b567c76e9" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057706 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf9a179-a462-4c72-aaf5-cd1af5717093" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057714 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c761ec4-d3b0-4663-88e7-2de45be4f9f7" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057721 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bd0e01-0001-426e-b7a3-e8140661136f" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057727 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd97a033-6160-4dc2-bde2-0ab637df4548" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057737 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4148eb-52e6-4c5b-ba64-3948bc5533ed" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057743 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf049fb-3442-41b6-abe4-b43daeaf22e2" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057750 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca06018-5e9c-4cc9-b360-9ead31686bf1" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057756 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48a647e-1b9e-4248-aa22-1349c838eb04" containerName="cleanup" Oct 30 22:15:55.057870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.057764 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="93159d6d-45b9-4b2c-8bcd-4dd2d3507b50" containerName="cleanup" Oct 30 22:15:55.063436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.063414 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.068230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.068203 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:55.068405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.068229 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:55.068405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.068315 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:15:55.068405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.068342 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:55.074046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.074020 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr"] Oct 30 22:15:55.144181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.144147 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.144398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.144194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsxt\" (UniqueName: \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.144398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.144222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.246153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.246121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.246385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.246170 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsxt\" (UniqueName: \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.246385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.246210 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.246918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.246893 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.248920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.248896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.258915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.258888 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsxt\" (UniqueName: \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\") pod \"maas-api-revoke-sub-33c92d3d-mvshr\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.401847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.401812 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:55.707861 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:55.707825 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5839a362_8356_4ce1_a2b9_ade8c5b39d82.slice/crio-6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2 WatchSource:0}: Error finding container 6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2: Status 404 returned error can't find the container with id 6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2 Oct 30 22:15:55.708711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:55.708677 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr"] Oct 30 22:15:56.238726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.238691 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:15:56.246647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.246625 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.259412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259385 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-e2e--96f1cc57-dockercfg-5bqlg\"" Oct 30 22:15:56.259562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259483 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"openshift-service-ca.crt\"" Oct 30 22:15:56.259562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259538 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"routing-overlay\"" Oct 30 22:15:56.259652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259605 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"kube-root-ca.crt\"" Oct 30 22:15:56.259687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259657 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-17e7bc508a71\"" Oct 30 22:15:56.259725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259706 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f8222\"" Oct 30 22:15:56.259781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.259769 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-cb4046ae8a86\"" Oct 30 22:15:56.260550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.260527 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:15:56.361289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.358516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdww\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.361289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.358736 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.361289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.358788 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.361289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.358820 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.461206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.461074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdww\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.462658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.461402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.462658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.461791 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.462658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.461846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.462658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.462601 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.462962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.462764 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.464824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.464798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.478223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.478195 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdww\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.594210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.594180 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:56.660018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.659009 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" event={"ID":"5839a362-8356-4ce1-a2b9-ade8c5b39d82","Type":"ContainerStarted","Data":"ffafdfa8379e1f4e4dbd80e9a251173b14b34af37cb06053aea3df73a888570a"} Oct 30 22:15:56.660018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.659075 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" event={"ID":"5839a362-8356-4ce1-a2b9-ade8c5b39d82","Type":"ContainerStarted","Data":"6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2"} Oct 30 22:15:56.981459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:56.981339 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" podStartSLOduration=1.981313286 podStartE2EDuration="1.981313286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:56.681514054 +0000 UTC m=+2180.631469000" watchObservedRunningTime="2026-10-30 22:15:56.981313286 +0000 UTC m=+2180.931268260" Oct 30 22:15:57.000893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.000858 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:15:57.268795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.268756 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:15:57.282249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.282222 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:15:57.282706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.282376 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.291835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.291816 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-277a231d8375\"" Oct 30 22:15:57.422759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.422726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.422759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.422762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.423027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.423003 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.423179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.423161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxg5t\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.525197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.525107 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.525396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.525292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg5t\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.525545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.525525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.525629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.525560 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.525893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.525866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.526512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.526474 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.528333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.528259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.542532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.542509 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg5t\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.557506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.557481 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:15:57.621292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.604026 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:57.699157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.693858 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" event={"ID":"0b4f6c4f-8742-4eb1-a391-0f5358ddc366","Type":"ContainerStarted","Data":"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56"} Oct 30 22:15:57.699157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.693928 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" event={"ID":"0b4f6c4f-8742-4eb1-a391-0f5358ddc366","Type":"ContainerStarted","Data":"b66f280116b2433e27e3d417ccd2babf5aa1a1f723dfde7e8b7068d367b72a50"} Oct 30 22:15:57.699157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.693971 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:15:57.703255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.702679 5036 generic.go:356] "Generic (PLEG): container finished" podID="5839a362-8356-4ce1-a2b9-ade8c5b39d82" containerID="ffafdfa8379e1f4e4dbd80e9a251173b14b34af37cb06053aea3df73a888570a" exitCode=0 Oct 30 22:15:57.703255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.702846 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" event={"ID":"5839a362-8356-4ce1-a2b9-ade8c5b39d82","Type":"ContainerDied","Data":"ffafdfa8379e1f4e4dbd80e9a251173b14b34af37cb06053aea3df73a888570a"} Oct 30 22:15:57.713826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.713769 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" podStartSLOduration=1.713747289 podStartE2EDuration="1.713747289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:57.712670578 +0000 UTC m=+2181.662625524" watchObservedRunningTime="2026-10-30 22:15:57.713747289 +0000 UTC m=+2181.663702237" Oct 30 22:15:57.982397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:57.982368 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:15:57.982581 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:15:57.982503 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8829a5c0_4ff6_4a56_9cf6_2478d88ee395.slice/crio-b9d8644b195f3f2651c277945031ec9b1c83f8dcf5497971bca508626bc17039 WatchSource:0}: Error finding container b9d8644b195f3f2651c277945031ec9b1c83f8dcf5497971bca508626bc17039: Status 404 returned error can't find the container with id b9d8644b195f3f2651c277945031ec9b1c83f8dcf5497971bca508626bc17039 Oct 30 22:15:58.497285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.497241 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:15:58.714883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.714847 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" event={"ID":"8829a5c0-4ff6-4a56-9cf6-2478d88ee395","Type":"ContainerStarted","Data":"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f"} Oct 30 22:15:58.715103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.714912 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" event={"ID":"8829a5c0-4ff6-4a56-9cf6-2478d88ee395","Type":"ContainerStarted","Data":"b9d8644b195f3f2651c277945031ec9b1c83f8dcf5497971bca508626bc17039"} Oct 30 22:15:58.715553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.715526 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:15:58.736221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.736170 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" podStartSLOduration=1.736151747 podStartE2EDuration="1.736151747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:58.733833304 +0000 UTC m=+2182.683788251" watchObservedRunningTime="2026-10-30 22:15:58.736151747 +0000 UTC m=+2182.686106694" Oct 30 22:15:58.969495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.969469 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:58.984615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.984585 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\" (UniqueName: \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\") pod \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " Oct 30 22:15:58.984856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.984839 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\") pod \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " Oct 30 22:15:58.984936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.984906 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\") pod \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\" (UID: \"5839a362-8356-4ce1-a2b9-ade8c5b39d82\") " Oct 30 22:15:58.985215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.985192 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca" pod "5839a362-8356-4ce1-a2b9-ade8c5b39d82" (UID: "5839a362-8356-4ce1-a2b9-ade8c5b39d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:58.985644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.985624 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.987379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.987349 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt" pod "5839a362-8356-4ce1-a2b9-ade8c5b39d82" (UID: "5839a362-8356-4ce1-a2b9-ade8c5b39d82"). InnerVolumeSpecName "kube-api-access-vbsxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:58.987787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:58.987762 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca" pod "5839a362-8356-4ce1-a2b9-ade8c5b39d82" (UID: "5839a362-8356-4ce1-a2b9-ade8c5b39d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:59.087677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:59.087595 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5839a362-8356-4ce1-a2b9-ade8c5b39d82-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:59.087677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:59.087624 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsxt\" (UniqueName: \"kubernetes.io/projected/5839a362-8356-4ce1-a2b9-ade8c5b39d82-kube-api-access-vbsxt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:59.729278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:59.729233 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" Oct 30 22:15:59.729278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:59.729252 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr" event={"ID":"5839a362-8356-4ce1-a2b9-ade8c5b39d82","Type":"ContainerDied","Data":"6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2"} Oct 30 22:15:59.729843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:15:59.729323 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff5a67e8f4a802702645cb2f7e0e0dd5289b85ff3dc14d99fb0a5097154d0d2" Oct 30 22:16:02.805653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.805615 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:16:02.846346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.846306 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:16:02.847783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.847757 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5839a362-8356-4ce1-a2b9-ade8c5b39d82" containerName="revoke-keys" Oct 30 22:16:02.847863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.847784 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5839a362-8356-4ce1-a2b9-ade8c5b39d82" containerName="revoke-keys" Oct 30 22:16:02.848073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.848060 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5839a362-8356-4ce1-a2b9-ade8c5b39d82" containerName="revoke-keys" Oct 30 22:16:02.852837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.852818 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:02.856631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.856608 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-33b3f9f9fd25\"" Oct 30 22:16:02.864121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.864095 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:16:02.978558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.978527 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:02.978779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.978600 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:02.978779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.978705 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbllt\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:02.978903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:02.978886 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.079966 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.080042 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.080069 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbllt\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.080211 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.080758 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.080966 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.080947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.082975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.082951 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.091917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.091894 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbllt\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.171436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.171397 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.515385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.515227 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:16:03.716135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.716092 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" podUID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" containerName="payload-processing" containerID="cri-o://4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56" gracePeriod=25 Oct 30 22:16:03.776916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.776840 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" event={"ID":"695563f5-e441-4538-b5b2-5617da6965b0","Type":"ContainerStarted","Data":"f2baa6bf5f7fa5c40a08cf54caf8de09724e806523e1f95de0a5579d01185716"} Oct 30 22:16:03.776916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.776895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" event={"ID":"695563f5-e441-4538-b5b2-5617da6965b0","Type":"ContainerStarted","Data":"ca2c0702dab4a003949840707111d271cbc8429d902a71331d06c1ef1c34b974"} Oct 30 22:16:03.777107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.776945 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:03.798557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:03.798510 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" podStartSLOduration=1.798497261 podStartE2EDuration="1.798497261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.794920424 +0000 UTC m=+2187.744875361" watchObservedRunningTime="2026-10-30 22:16:03.798497261 +0000 UTC m=+2187.748452207" Oct 30 22:16:04.004313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.004287 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:16:04.097948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.097815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\") pod \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " Oct 30 22:16:04.099849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.099819 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume" pod "0b4f6c4f-8742-4eb1-a391-0f5358ddc366" (UID: "0b4f6c4f-8742-4eb1-a391-0f5358ddc366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:04.100127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.100106 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\") pod \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " Oct 30 22:16:04.100350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.100328 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\") pod \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " Oct 30 22:16:04.100449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.100397 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\") pod \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\" (UID: \"0b4f6c4f-8742-4eb1-a391-0f5358ddc366\") " Oct 30 22:16:04.101080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.101052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume" pod "0b4f6c4f-8742-4eb1-a391-0f5358ddc366" (UID: "0b4f6c4f-8742-4eb1-a391-0f5358ddc366"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:04.102055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.102034 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:04.102162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.102058 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:04.104200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.104176 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww" pod "0b4f6c4f-8742-4eb1-a391-0f5358ddc366" (UID: "0b4f6c4f-8742-4eb1-a391-0f5358ddc366"). InnerVolumeSpecName "kube-api-access-8mdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:04.104356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.104210 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials" pod "0b4f6c4f-8742-4eb1-a391-0f5358ddc366" (UID: "0b4f6c4f-8742-4eb1-a391-0f5358ddc366"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:04.218722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.212782 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mdww\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-kube-api-access-8mdww\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:04.218722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.212821 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b4f6c4f-8742-4eb1-a391-0f5358ddc366-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:04.790948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.790915 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" containerID="4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56" exitCode=0 Oct 30 22:16:04.791140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.790963 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" Oct 30 22:16:04.791140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.791003 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" event={"ID":"0b4f6c4f-8742-4eb1-a391-0f5358ddc366","Type":"ContainerDied","Data":"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56"} Oct 30 22:16:04.791140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.791067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc" event={"ID":"0b4f6c4f-8742-4eb1-a391-0f5358ddc366","Type":"ContainerDied","Data":"b66f280116b2433e27e3d417ccd2babf5aa1a1f723dfde7e8b7068d367b72a50"} Oct 30 22:16:04.791140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.791092 5036 scope.go:122] "RemoveContainer" containerID="4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56" Oct 30 22:16:04.803790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.803770 5036 scope.go:122] "RemoveContainer" containerID="4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56" Oct 30 22:16:04.804232 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:04.804201 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56\": container with ID starting with 4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56 not found: ID does not exist" containerID="4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56" Oct 30 22:16:04.804380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.804253 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56"} err="failed to get container status \"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56\": rpc error: code = NotFound desc = could not find container \"4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56\": container with ID starting with 4eed942fe4df680479305a47caa22fd473fcdef58d1a5d3069575b5b509b8c56 not found: ID does not exist" Oct 30 22:16:04.817032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.817006 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:16:04.824821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.824792 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4fc5jtc"] Oct 30 22:16:04.910288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.907296 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:16:04.910288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.908769 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" containerName="payload-processing" Oct 30 22:16:04.910288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.908795 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" containerName="payload-processing" Oct 30 22:16:04.910288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.909072 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" containerName="payload-processing" Oct 30 22:16:04.946288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.941863 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:04.960834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.959982 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a39158-openshift-default-dockercfg-xz2kf\"" Oct 30 22:16:04.978300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:04.976952 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:16:05.032259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032204 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2c9d\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032417 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032585 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032613 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032673 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.032786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032716 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.037358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032791 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.037358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.032835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134280 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c9d\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134323 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134434 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134521 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134565 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134640 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.134801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.134684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.135130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.135107 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.135203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.135178 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.135287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.135159 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.135357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.135307 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.135416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.135383 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.137127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.137104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.138755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.138727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.148077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.148050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.148329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.148260 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c9d\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.288428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.288378 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:05.597329 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:05.597299 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24c126b_e324_4a50_92e4_318799ce2870.slice/crio-1d0ebcde7d2f7b53fefa19325ac1be9c2f4877f529774c58db4d6d4fc0ecae32 WatchSource:0}: Error finding container 1d0ebcde7d2f7b53fefa19325ac1be9c2f4877f529774c58db4d6d4fc0ecae32: Status 404 returned error can't find the container with id 1d0ebcde7d2f7b53fefa19325ac1be9c2f4877f529774c58db4d6d4fc0ecae32 Oct 30 22:16:05.599164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.599137 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:16:05.601022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.600988 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:05.601106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.601085 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:05.601146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.601117 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:05.838186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.838145 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" event={"ID":"f24c126b-e324-4a50-92e4-318799ce2870","Type":"ContainerStarted","Data":"e8140be39060ee801ea85ad496afd715865e5485efc02e81849181e889c828e7"} Oct 30 22:16:05.838533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.838510 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" event={"ID":"f24c126b-e324-4a50-92e4-318799ce2870","Type":"ContainerStarted","Data":"1d0ebcde7d2f7b53fefa19325ac1be9c2f4877f529774c58db4d6d4fc0ecae32"} Oct 30 22:16:05.877385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.877221 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podStartSLOduration=1.8772018510000001 podStartE2EDuration="1.877201851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:05.871187998 +0000 UTC m=+2189.821142985" watchObservedRunningTime="2026-10-30 22:16:05.877201851 +0000 UTC m=+2189.827156797" Oct 30 22:16:05.992198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:05.992161 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:16:06.282120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:06.282078 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4f6c4f-8742-4eb1-a391-0f5358ddc366" path="/var/lib/kubelet/pods/0b4f6c4f-8742-4eb1-a391-0f5358ddc366/volumes" Oct 30 22:16:06.305298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:06.291196 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:06.305596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:06.292797 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.305767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:06.305740 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:07.289689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:07.289651 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.290179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:07.289708 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:07.806592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:07.806549 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" podUID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" containerName="payload-processing" containerID="cri-o://88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f" gracePeriod=25 Oct 30 22:16:08.256014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.255973 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.281218 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\") pod \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.281430 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\") pod \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.281478 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\") pod \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.281545 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\") pod \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\" (UID: \"8829a5c0-4ff6-4a56-9cf6-2478d88ee395\") " Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.282698 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume" pod "8829a5c0-4ff6-4a56-9cf6-2478d88ee395" (UID: "8829a5c0-4ff6-4a56-9cf6-2478d88ee395"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.283079 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume" pod "8829a5c0-4ff6-4a56-9cf6-2478d88ee395" (UID: "8829a5c0-4ff6-4a56-9cf6-2478d88ee395"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.283739 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:08.284290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.283767 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:08.299287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.289719 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.299287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.289776 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:08.299287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.290365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t" pod "8829a5c0-4ff6-4a56-9cf6-2478d88ee395" (UID: "8829a5c0-4ff6-4a56-9cf6-2478d88ee395"). InnerVolumeSpecName "kube-api-access-xxg5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:08.299287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.293412 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials" pod "8829a5c0-4ff6-4a56-9cf6-2478d88ee395" (UID: "8829a5c0-4ff6-4a56-9cf6-2478d88ee395"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:08.389305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.387874 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxg5t\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-kube-api-access-xxg5t\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:08.389305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.387909 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8829a5c0-4ff6-4a56-9cf6-2478d88ee395-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:08.942077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.931465 5036 generic.go:356] "Generic (PLEG): container finished" podID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" containerID="88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f" exitCode=0 Oct 30 22:16:08.942077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.931840 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" event={"ID":"8829a5c0-4ff6-4a56-9cf6-2478d88ee395","Type":"ContainerDied","Data":"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f"} Oct 30 22:16:08.942077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.931898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" event={"ID":"8829a5c0-4ff6-4a56-9cf6-2478d88ee395","Type":"ContainerDied","Data":"b9d8644b195f3f2651c277945031ec9b1c83f8dcf5497971bca508626bc17039"} Oct 30 22:16:08.942077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.931929 5036 scope.go:122] "RemoveContainer" containerID="88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f" Oct 30 22:16:08.942077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.932096 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5" Oct 30 22:16:08.989500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.988740 5036 scope.go:122] "RemoveContainer" containerID="88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f" Oct 30 22:16:08.989500 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:08.989161 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f\": container with ID starting with 88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f not found: ID does not exist" containerID="88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f" Oct 30 22:16:08.989500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.989195 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f"} err="failed to get container status \"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f\": rpc error: code = NotFound desc = could not find container \"88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f\": container with ID starting with 88e82d8c101270e49db5671fe1220e6ee2b9ab10152df1e3ec107bbb15f6295f not found: ID does not exist" Oct 30 22:16:08.993428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:08.993401 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:16:09.000747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:09.000724 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c594zbl5"] Oct 30 22:16:09.049180 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:09.049124 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8829a5c0_4ff6_4a56_9cf6_2478d88ee395.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:09.303349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:09.294648 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:09.303349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:09.294729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:10.191770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:10.191736 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" path="/var/lib/kubelet/pods/8829a5c0-4ff6-4a56-9cf6-2478d88ee395/volumes" Oct 30 22:16:10.288789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:10.288749 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:10.289004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:10.288813 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:10.998399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:10.998352 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" podUID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" containerName="payload-processing" containerID="cri-o://aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f" gracePeriod=25 Oct 30 22:16:11.292357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.289809 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:11.292357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.289888 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:11.700227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.700202 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:16:11.854043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.853962 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\") pod \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " Oct 30 22:16:11.854043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.854002 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\") pod \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " Oct 30 22:16:11.854276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.854248 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\") pod \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " Oct 30 22:16:11.854339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.854303 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\") pod \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\" (UID: \"585c658b-ba8c-4fc6-a1c8-16f55d4519e1\") " Oct 30 22:16:11.854481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.854445 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume" pod "585c658b-ba8c-4fc6-a1c8-16f55d4519e1" (UID: "585c658b-ba8c-4fc6-a1c8-16f55d4519e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:11.854741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.854709 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume" pod "585c658b-ba8c-4fc6-a1c8-16f55d4519e1" (UID: "585c658b-ba8c-4fc6-a1c8-16f55d4519e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:11.855164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.855145 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:11.855164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.855169 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:11.856794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.856766 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials" pod "585c658b-ba8c-4fc6-a1c8-16f55d4519e1" (UID: "585c658b-ba8c-4fc6-a1c8-16f55d4519e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:11.856891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.856804 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc" pod "585c658b-ba8c-4fc6-a1c8-16f55d4519e1" (UID: "585c658b-ba8c-4fc6-a1c8-16f55d4519e1"). InnerVolumeSpecName "kube-api-access-7wbsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:11.965291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.961555 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:11.965291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:11.961598 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wbsc\" (UniqueName: \"kubernetes.io/projected/585c658b-ba8c-4fc6-a1c8-16f55d4519e1-kube-api-access-7wbsc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.081797 5036 generic.go:356] "Generic (PLEG): container finished" podID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" containerID="aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f" exitCode=0 Oct 30 22:16:12.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.081853 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" Oct 30 22:16:12.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.081861 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" event={"ID":"585c658b-ba8c-4fc6-a1c8-16f55d4519e1","Type":"ContainerDied","Data":"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f"} Oct 30 22:16:12.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.082022 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x" event={"ID":"585c658b-ba8c-4fc6-a1c8-16f55d4519e1","Type":"ContainerDied","Data":"efe635a1024965d2d6dc8ede3af12f53451ed3267552765c4090c49d8aeb49e2"} Oct 30 22:16:12.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.082055 5036 scope.go:122] "RemoveContainer" containerID="aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f" Oct 30 22:16:12.097731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.097711 5036 scope.go:122] "RemoveContainer" containerID="aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f" Oct 30 22:16:12.098070 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:12.098051 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f\": container with ID starting with aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f not found: ID does not exist" containerID="aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f" Oct 30 22:16:12.098116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.098078 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f"} err="failed to get container status \"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f\": rpc error: code = NotFound desc = could not find container \"aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f\": container with ID starting with aad2b0c0bda62742e738b7f4e6483bb291abd1ac83cfb848c773c38d5ec6b56f not found: ID does not exist" Oct 30 22:16:12.115339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.110246 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:16:12.149731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.149692 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-8dc544s5x"] Oct 30 22:16:12.191789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.191759 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" path="/var/lib/kubelet/pods/585c658b-ba8c-4fc6-a1c8-16f55d4519e1/volumes" Oct 30 22:16:12.289564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.289527 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:12.289759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:12.289595 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 22:16:13.744524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:13.744485 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:13.745181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:13.745007 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:13.748027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:13.748005 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:16:14.795138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:14.795106 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:16:14.864335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:14.862510 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:16:18.102350 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:18.102252 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:16:19.863310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:19.863218 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" podUID="30d78f7b-a040-45ae-9e84-e40ac20bd801" containerName="payload-processing" containerID="cri-o://131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a" gracePeriod=25 Oct 30 22:16:20.183753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.183720 5036 generic.go:356] "Generic (PLEG): container finished" podID="30d78f7b-a040-45ae-9e84-e40ac20bd801" containerID="131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a" exitCode=0 Oct 30 22:16:20.184102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.184081 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:16:20.191910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.191884 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" event={"ID":"30d78f7b-a040-45ae-9e84-e40ac20bd801","Type":"ContainerDied","Data":"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a"} Oct 30 22:16:20.192035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.191930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" event={"ID":"30d78f7b-a040-45ae-9e84-e40ac20bd801","Type":"ContainerDied","Data":"787f0be79aba3e103350cb1030f23c9968236bfd9e6b4786caa7706661528c45"} Oct 30 22:16:20.192035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.191953 5036 scope.go:122] "RemoveContainer" containerID="131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a" Oct 30 22:16:20.207805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.207777 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\") pod \"30d78f7b-a040-45ae-9e84-e40ac20bd801\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " Oct 30 22:16:20.207962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.207941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\") pod \"30d78f7b-a040-45ae-9e84-e40ac20bd801\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " Oct 30 22:16:20.208042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.208027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\") pod \"30d78f7b-a040-45ae-9e84-e40ac20bd801\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " Oct 30 22:16:20.208111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.208056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\") pod \"30d78f7b-a040-45ae-9e84-e40ac20bd801\" (UID: \"30d78f7b-a040-45ae-9e84-e40ac20bd801\") " Oct 30 22:16:20.209332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.209311 5036 scope.go:122] "RemoveContainer" containerID="131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a" Oct 30 22:16:20.211766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.210239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume" pod "30d78f7b-a040-45ae-9e84-e40ac20bd801" (UID: "30d78f7b-a040-45ae-9e84-e40ac20bd801"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:20.211766 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:20.210414 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a\": container with ID starting with 131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a not found: ID does not exist" containerID="131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a" Oct 30 22:16:20.211766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.210453 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a"} err="failed to get container status \"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a\": rpc error: code = NotFound desc = could not find container \"131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a\": container with ID starting with 131b7bd0a1a3161c955d1128cfdc46d1ee3695cd1515cd11e8186ebef8c43a7a not found: ID does not exist" Oct 30 22:16:20.211766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.211010 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume" pod "30d78f7b-a040-45ae-9e84-e40ac20bd801" (UID: "30d78f7b-a040-45ae-9e84-e40ac20bd801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:20.212577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.212541 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln" pod "30d78f7b-a040-45ae-9e84-e40ac20bd801" (UID: "30d78f7b-a040-45ae-9e84-e40ac20bd801"). InnerVolumeSpecName "kube-api-access-f9mln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:20.213288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.213239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials" pod "30d78f7b-a040-45ae-9e84-e40ac20bd801" (UID: "30d78f7b-a040-45ae-9e84-e40ac20bd801"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:20.310508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.310470 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mln\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-kube-api-access-f9mln\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:20.310508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.310505 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:20.310716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.310521 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d78f7b-a040-45ae-9e84-e40ac20bd801-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:20.310716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:20.310535 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30d78f7b-a040-45ae-9e84-e40ac20bd801-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:21.193986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:21.193955 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm" Oct 30 22:16:21.304332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:21.300815 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:16:21.307728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:21.307697 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd82l9sm"] Oct 30 22:16:22.194012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:22.193978 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d78f7b-a040-45ae-9e84-e40ac20bd801" path="/var/lib/kubelet/pods/30d78f7b-a040-45ae-9e84-e40ac20bd801/volumes" Oct 30 22:16:27.188548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:27.188513 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 22:16:27.189027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:27.188718 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" containerID="cri-o://a1a20b3e9f2cd6ee0cae2229b875ccc4ee8822d934e111f8b5288f6fcb0f7896" gracePeriod=30 Oct 30 22:16:28.569040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:28.569002 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:16:28.569624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:28.569237 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" containerID="cri-o://a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc" gracePeriod=30 Oct 30 22:16:28.574926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:28.574899 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:16:28.575124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:28.575087 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" containerID="cri-o://4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9" gracePeriod=30 Oct 30 22:16:30.934058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:30.934016 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:16:30.934609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:30.934295 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" containerID="cri-o://5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c" gracePeriod=30 Oct 30 22:16:32.488202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.488157 5036 generic.go:356] "Generic (PLEG): container finished" podID="4208bd2e-e746-4e58-afc9-058846b00e29" containerID="a1a20b3e9f2cd6ee0cae2229b875ccc4ee8822d934e111f8b5288f6fcb0f7896" exitCode=0 Oct 30 22:16:32.488775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.488224 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" event={"ID":"4208bd2e-e746-4e58-afc9-058846b00e29","Type":"ContainerDied","Data":"a1a20b3e9f2cd6ee0cae2229b875ccc4ee8822d934e111f8b5288f6fcb0f7896"} Oct 30 22:16:32.488775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.488324 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" event={"ID":"4208bd2e-e746-4e58-afc9-058846b00e29","Type":"ContainerDied","Data":"f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd"} Oct 30 22:16:32.488775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.488357 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e395e4cc2315fc9c81d15e6c348c2e3df6cfc3bdfc7d6664de0926455514bd" Oct 30 22:16:32.506337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.506309 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 22:16:32.606068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606006 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606074 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606135 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606235 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606299 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606429 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606608 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606712 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.606849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.606825 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\") pod \"4208bd2e-e746-4e58-afc9-058846b00e29\" (UID: \"4208bd2e-e746-4e58-afc9-058846b00e29\") " Oct 30 22:16:32.607949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.607761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:32.608416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.607594 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:32.609062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.609015 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:32.611532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.611330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:32.612029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.611755 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:32.613572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.613540 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:32.622723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.622690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:32.622833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.622807 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:32.622954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.622931 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz" pod "4208bd2e-e746-4e58-afc9-058846b00e29" (UID: "4208bd2e-e746-4e58-afc9-058846b00e29"). InnerVolumeSpecName "kube-api-access-dzhlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:32.709078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709036 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4208bd2e-e746-4e58-afc9-058846b00e29-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709070 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhlz\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-kube-api-access-dzhlz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709084 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709094 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709105 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709116 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4208bd2e-e746-4e58-afc9-058846b00e29-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709126 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709134 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4208bd2e-e746-4e58-afc9-058846b00e29-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.709480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:32.709145 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4208bd2e-e746-4e58-afc9-058846b00e29-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.421352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.421311 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:16:33.458871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.458836 5036 cpu_manager.go:414] "containerMap: removing container" podUID="30d78f7b-a040-45ae-9e84-e40ac20bd801" containerName="payload-processing" Oct 30 22:16:33.459106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.459091 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d78f7b-a040-45ae-9e84-e40ac20bd801" containerName="payload-processing" Oct 30 22:16:33.459212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.459202 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" containerName="payload-processing" Oct 30 22:16:33.517647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.517610 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" containerName="payload-processing" Oct 30 22:16:33.530423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.530384 5036 cpu_manager.go:414] "containerMap: removing container" podUID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" containerName="payload-processing" Oct 30 22:16:33.530654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.530637 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" containerName="payload-processing" Oct 30 22:16:33.530784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.530773 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" Oct 30 22:16:33.530870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.530859 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" Oct 30 22:16:33.531362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.531338 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d78f7b-a040-45ae-9e84-e40ac20bd801" containerName="payload-processing" Oct 30 22:16:33.593407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.593359 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8829a5c0-4ff6-4a56-9cf6-2478d88ee395" containerName="payload-processing" Oct 30 22:16:33.593742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.593721 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="585c658b-ba8c-4fc6-a1c8-16f55d4519e1" containerName="payload-processing" Oct 30 22:16:33.638969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.608837 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" containerName="istio-proxy" Oct 30 22:16:33.685349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.681638 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv" Oct 30 22:16:33.736370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.736339 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:16:33.736534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.736475 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.744777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.744746 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-zdlx9\"" Oct 30 22:16:33.744894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.744832 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-813840a060cc\"" Oct 30 22:16:33.744945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.744920 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\"" Oct 30 22:16:33.745529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.744976 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\"" Oct 30 22:16:33.745648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.745625 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\"" Oct 30 22:16:33.746708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.745709 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\"" Oct 30 22:16:33.803415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.803252 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 22:16:33.824285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.824225 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-v8kmv"] Oct 30 22:16:33.864364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.852674 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.864364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.852890 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.864364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.853080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxqm\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.864364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.853186 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.963699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.963594 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.980199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.980156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.980713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.980685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxqm\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.980977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.980956 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.981500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.981059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.994746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.982149 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:33.994746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:33.994472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:34.008308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.006933 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxqm\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:34.070598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.070546 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:34.170709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.170258 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:16:34.235981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.235912 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:16:34.238234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.238205 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4208bd2e-e746-4e58-afc9-058846b00e29" path="/var/lib/kubelet/pods/4208bd2e-e746-4e58-afc9-058846b00e29/volumes" Oct 30 22:16:34.293048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.293014 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.293938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.293906 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.294333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.294279 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.294519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.294355 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.294882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.294858 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.295498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.295476 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.296160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.295744 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.296423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.296406 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.296916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.296893 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.297615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.297465 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:34.297961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.297940 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.298481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.298361 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.298669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.298651 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\") pod \"4824aafc-23a1-4d71-9318-ba472a4f044e\" (UID: \"4824aafc-23a1-4d71-9318-ba472a4f044e\") " Oct 30 22:16:34.299621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.299059 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.299621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.299476 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "kube-api-access-dt488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.299621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.299583 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.299848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.299648 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:34.300510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300486 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300566 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300589 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300605 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt488\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-kube-api-access-dt488\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300621 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300637 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300652 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824aafc-23a1-4d71-9318-ba472a4f044e-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300665 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.300944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.300764 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy" pod "4824aafc-23a1-4d71-9318-ba472a4f044e" (UID: "4824aafc-23a1-4d71-9318-ba472a4f044e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.401168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401134 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401195 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401253 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401416 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401467 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.401515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401547 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401669 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.401688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401681 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401745 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.401900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401853 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.401900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401892 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\") pod \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\" (UID: \"2b70e666-6ed9-4677-8a0b-6598bbd4c28c\") " Oct 30 22:16:34.402019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.402019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.401944 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:34.402893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402864 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.403158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402899 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.403158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402917 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.403158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402933 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.403158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402948 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824aafc-23a1-4d71-9318-ba472a4f044e-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.403158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.402964 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.404520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.404500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.404766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.404744 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:34.404829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.404794 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "kube-api-access-5pcbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.404829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.404794 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy" pod "2b70e666-6ed9-4677-8a0b-6598bbd4c28c" (UID: "2b70e666-6ed9-4677-8a0b-6598bbd4c28c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:34.415225 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:34.415205 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb525a859_41f6_4667_b69d_6cfbb54e03e4.slice/crio-83df9c167b3cb86b00cd00972a5274fb65a0f8cc70a134bb1a4100bb85e848fa WatchSource:0}: Error finding container 83df9c167b3cb86b00cd00972a5274fb65a0f8cc70a134bb1a4100bb85e848fa: Status 404 returned error can't find the container with id 83df9c167b3cb86b00cd00972a5274fb65a0f8cc70a134bb1a4100bb85e848fa Oct 30 22:16:34.418766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.418741 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:16:34.504195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.504152 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.504327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.504203 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcbw\" (UniqueName: \"kubernetes.io/projected/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-kube-api-access-5pcbw\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.504327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.504218 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.504327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.504235 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b70e666-6ed9-4677-8a0b-6598bbd4c28c-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.693766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.693729 5036 generic.go:356] "Generic (PLEG): container finished" podID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerID="4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9" exitCode=0 Oct 30 22:16:34.694335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.693811 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" Oct 30 22:16:34.694335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.693815 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" event={"ID":"2b70e666-6ed9-4677-8a0b-6598bbd4c28c","Type":"ContainerDied","Data":"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9"} Oct 30 22:16:34.694335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.693880 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw" event={"ID":"2b70e666-6ed9-4677-8a0b-6598bbd4c28c","Type":"ContainerDied","Data":"9c0b988cee0374da5fde4c1aa8fe25300d3e5052b1b5f2aa81cd0204ddcd0d56"} Oct 30 22:16:34.694335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.693905 5036 scope.go:122] "RemoveContainer" containerID="4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9" Oct 30 22:16:34.696435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.696404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" event={"ID":"b525a859-41f6-4667-b69d-6cfbb54e03e4","Type":"ContainerStarted","Data":"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457"} Oct 30 22:16:34.696598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.696474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" event={"ID":"b525a859-41f6-4667-b69d-6cfbb54e03e4","Type":"ContainerStarted","Data":"83df9c167b3cb86b00cd00972a5274fb65a0f8cc70a134bb1a4100bb85e848fa"} Oct 30 22:16:34.696598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.696509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:34.698404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.698375 5036 generic.go:356] "Generic (PLEG): container finished" podID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerID="a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc" exitCode=0 Oct 30 22:16:34.698531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.698461 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" event={"ID":"4824aafc-23a1-4d71-9318-ba472a4f044e","Type":"ContainerDied","Data":"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc"} Oct 30 22:16:34.698531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.698518 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" event={"ID":"4824aafc-23a1-4d71-9318-ba472a4f044e","Type":"ContainerDied","Data":"ab9be92e1dce9ff99df96cfcf0e1391f633b20f506709c92601c71129eae7af2"} Oct 30 22:16:34.698658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.698443 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8" Oct 30 22:16:34.708422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.708404 5036 scope.go:122] "RemoveContainer" containerID="4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9" Oct 30 22:16:34.708792 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:34.708759 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9\": container with ID starting with 4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9 not found: ID does not exist" containerID="4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9" Oct 30 22:16:34.708893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.708796 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9"} err="failed to get container status \"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9\": rpc error: code = NotFound desc = could not find container \"4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9\": container with ID starting with 4d03f550a19ac46ed9d37f83cb0f345a8f502b014b98173e94ed1dfef6dbcba9 not found: ID does not exist" Oct 30 22:16:34.708893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.708825 5036 scope.go:122] "RemoveContainer" containerID="a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc" Oct 30 22:16:34.722842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.722816 5036 scope.go:122] "RemoveContainer" containerID="a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc" Oct 30 22:16:34.723361 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:34.723331 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc\": container with ID starting with a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc not found: ID does not exist" containerID="a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc" Oct 30 22:16:34.723446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.723372 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc"} err="failed to get container status \"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc\": rpc error: code = NotFound desc = could not find container \"a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc\": container with ID starting with a753ff8df84364d10a2491647b89b948db4e6e7b66543176fae26644462a87dc not found: ID does not exist" Oct 30 22:16:34.723645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.723611 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" podStartSLOduration=1.723599563 podStartE2EDuration="1.723599563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:34.717286939 +0000 UTC m=+2218.667241892" watchObservedRunningTime="2026-10-30 22:16:34.723599563 +0000 UTC m=+2218.673554515" Oct 30 22:16:34.739351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.739323 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:16:34.743679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.743659 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-d2rd8"] Oct 30 22:16:34.754204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.754138 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:16:34.758944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:34.758909 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-bcvkw"] Oct 30 22:16:36.217542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.204048 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" path="/var/lib/kubelet/pods/2b70e666-6ed9-4677-8a0b-6598bbd4c28c/volumes" Oct 30 22:16:36.253495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.253459 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" path="/var/lib/kubelet/pods/4824aafc-23a1-4d71-9318-ba472a4f044e/volumes" Oct 30 22:16:36.331067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.331036 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:16:36.356605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.356573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.356952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.356920 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:36.357085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.356943 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357230 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357378 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357546 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357674 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.357975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357959 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.358190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.358173 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\") pod \"38d17b81-8f70-432d-81a0-4b18db51fd3b\" (UID: \"38d17b81-8f70-432d-81a0-4b18db51fd3b\") " Oct 30 22:16:36.358925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357736 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:36.358925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.358883 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:36.358925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.357960 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:36.358925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.358697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:36.359315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.359245 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.359315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.359290 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d17b81-8f70-432d-81a0-4b18db51fd3b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.359315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.359306 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.359524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.359319 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.359524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.359333 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.362234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.362169 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "kube-api-access-n7tr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:36.362493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.362475 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:36.364817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.364409 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:36.365490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.365464 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token" pod "38d17b81-8f70-432d-81a0-4b18db51fd3b" (UID: "38d17b81-8f70-432d-81a0-4b18db51fd3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:36.460819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.460787 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.460819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.460817 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.461047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.460829 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7tr4\" (UniqueName: \"kubernetes.io/projected/38d17b81-8f70-432d-81a0-4b18db51fd3b-kube-api-access-n7tr4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.461047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.460845 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d17b81-8f70-432d-81a0-4b18db51fd3b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.723254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.723213 5036 generic.go:356] "Generic (PLEG): container finished" podID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerID="5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c" exitCode=0 Oct 30 22:16:36.723453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.723295 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" Oct 30 22:16:36.723453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.723300 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerDied","Data":"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c"} Oct 30 22:16:36.723453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.723358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9" event={"ID":"38d17b81-8f70-432d-81a0-4b18db51fd3b","Type":"ContainerDied","Data":"d670128767d09566e80eec84ac0b58d7b65ac0e3fa58c0446a32de23555f4e69"} Oct 30 22:16:36.723453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.723380 5036 scope.go:122] "RemoveContainer" containerID="5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c" Oct 30 22:16:36.736792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.736769 5036 scope.go:122] "RemoveContainer" containerID="57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e" Oct 30 22:16:36.749920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.749894 5036 scope.go:122] "RemoveContainer" containerID="5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c" Oct 30 22:16:36.750303 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:36.750239 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c\": container with ID starting with 5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c not found: ID does not exist" containerID="5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c" Oct 30 22:16:36.750363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.750318 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c"} err="failed to get container status \"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c\": rpc error: code = NotFound desc = could not find container \"5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c\": container with ID starting with 5883d440b292f9409b4009fa485a464dfa093eb608e638d3003be96b6953c81c not found: ID does not exist" Oct 30 22:16:36.750363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.750338 5036 scope.go:122] "RemoveContainer" containerID="57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e" Oct 30 22:16:36.750639 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:36.750611 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e\": container with ID starting with 57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e not found: ID does not exist" containerID="57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e" Oct 30 22:16:36.750730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.750642 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e"} err="failed to get container status \"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e\": rpc error: code = NotFound desc = could not find container \"57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e\": container with ID starting with 57fbdc7e7c1d635f9d71541ad07f54e9391cc6d0a267467b1230216b1e3b228e not found: ID does not exist" Oct 30 22:16:36.751656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.751635 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:16:36.758820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:36.758797 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-5tlz9"] Oct 30 22:16:38.191188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:38.191156 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" path="/var/lib/kubelet/pods/38d17b81-8f70-432d-81a0-4b18db51fd3b/volumes" Oct 30 22:16:39.784818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:39.784784 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 22:16:39.785688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:39.785035 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" containerID="cri-o://b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89" gracePeriod=30 Oct 30 22:16:39.792342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:39.792318 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 22:16:39.792559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:39.792538 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" containerID="cri-o://11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301" gracePeriod=30 Oct 30 22:16:40.202964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:40.202933 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 22:16:40.203143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:40.203123 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" containerID="cri-o://4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be" gracePeriod=30 Oct 30 22:16:41.321848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.321800 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn"] Oct 30 22:16:41.323640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323587 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" Oct 30 22:16:41.323640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323613 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323654 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323662 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323689 5036 cpu_manager.go:414] "containerMap: removing container" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323698 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323714 5036 cpu_manager.go:414] "containerMap: removing container" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:41.323824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323721 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:41.324154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323930 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4824aafc-23a1-4d71-9318-ba472a4f044e" containerName="istio-proxy" Oct 30 22:16:41.324154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323949 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:41.324154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.323971 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b70e666-6ed9-4677-8a0b-6598bbd4c28c" containerName="istio-proxy" Oct 30 22:16:41.329970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.329944 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.334775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.334671 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:41.334775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.334694 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:16:41.335370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.335330 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:41.336779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.336760 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:41.342325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.341517 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn"] Oct 30 22:16:41.478347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.478311 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.478525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.478437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8fl\" (UniqueName: \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.478525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.478495 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.580789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.580705 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.580789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.580789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8fl\" (UniqueName: \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.581014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.580835 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.581597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.581573 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.583489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.583465 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.592720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.592702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8fl\" (UniqueName: \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\") pod \"maas-api-revoke-sub-d818aa0c-gt8bn\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.658136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.658084 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:41.954354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.954221 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 22:16:41.954560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.954530 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" containerID="cri-o://60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a" gracePeriod=30 Oct 30 22:16:41.961087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:41.961048 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn"] Oct 30 22:16:41.962535 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:41.962504 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb3042b_235e_4733_ab8e_4552b525905d.slice/crio-3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f WatchSource:0}: Error finding container 3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f: Status 404 returned error can't find the container with id 3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f Oct 30 22:16:42.791770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:42.791664 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" event={"ID":"9fb3042b-235e-4733-ab8e-4552b525905d","Type":"ContainerStarted","Data":"3b849766b583a273985dd972b4847c98ffe8c9a7e4ed5044e7201ad13fd4f16c"} Oct 30 22:16:42.791770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:42.791765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" event={"ID":"9fb3042b-235e-4733-ab8e-4552b525905d","Type":"ContainerStarted","Data":"3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f"} Oct 30 22:16:42.811474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:42.811414 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" podStartSLOduration=1.811397109 podStartE2EDuration="1.811397109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:42.808421385 +0000 UTC m=+2226.758376343" watchObservedRunningTime="2026-10-30 22:16:42.811397109 +0000 UTC m=+2226.761352057" Oct 30 22:16:43.803615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:43.803581 5036 generic.go:356] "Generic (PLEG): container finished" podID="9fb3042b-235e-4733-ab8e-4552b525905d" containerID="3b849766b583a273985dd972b4847c98ffe8c9a7e4ed5044e7201ad13fd4f16c" exitCode=0 Oct 30 22:16:43.804019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:43.803659 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" event={"ID":"9fb3042b-235e-4733-ab8e-4552b525905d","Type":"ContainerDied","Data":"3b849766b583a273985dd972b4847c98ffe8c9a7e4ed5044e7201ad13fd4f16c"} Oct 30 22:16:44.067386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:44.063540 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:44.067386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:44.063609 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:45.080182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.080151 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:45.166278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.166233 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\") pod \"9fb3042b-235e-4733-ab8e-4552b525905d\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " Oct 30 22:16:45.166496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.166475 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\") pod \"9fb3042b-235e-4733-ab8e-4552b525905d\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " Oct 30 22:16:45.166683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.166664 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\" (UniqueName: \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\") pod \"9fb3042b-235e-4733-ab8e-4552b525905d\" (UID: \"9fb3042b-235e-4733-ab8e-4552b525905d\") " Oct 30 22:16:45.166771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.166688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca" pod "9fb3042b-235e-4733-ab8e-4552b525905d" (UID: "9fb3042b-235e-4733-ab8e-4552b525905d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:45.167434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.167357 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.170654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.170616 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl" pod "9fb3042b-235e-4733-ab8e-4552b525905d" (UID: "9fb3042b-235e-4733-ab8e-4552b525905d"). InnerVolumeSpecName "kube-api-access-6b8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.172925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.172901 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca" pod "9fb3042b-235e-4733-ab8e-4552b525905d" (UID: "9fb3042b-235e-4733-ab8e-4552b525905d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:45.202106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.201760 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 22:16:45.270881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.270845 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.270907 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.270947 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271015 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271042 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271202 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271316 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271359 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.271653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271403 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\") pod \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\" (UID: \"194dc5f4-757c-422b-a4ee-ac0482d8b0e4\") " Oct 30 22:16:45.272017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271975 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb3042b-235e-4733-ab8e-4552b525905d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.272017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.271989 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8fl\" (UniqueName: \"kubernetes.io/projected/9fb3042b-235e-4733-ab8e-4552b525905d-kube-api-access-6b8fl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.274075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.274042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.274242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.274229 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.274426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.274410 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.274750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.274728 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.274928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.274798 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:45.277407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.277382 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.277825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.277800 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "kube-api-access-g48w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.278446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.278424 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.279554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.279531 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo" pod "194dc5f4-757c-422b-a4ee-ac0482d8b0e4" (UID: "194dc5f4-757c-422b-a4ee-ac0482d8b0e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:45.285417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.285395 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 22:16:45.373064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.372980 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.373337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.373315 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.373573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.373451 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.373904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.373868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.374014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.373524 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.374116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374106 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.374249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374232 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.374450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.374698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.374847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374830 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.375037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.374996 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:45.375195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.375108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.375195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.375161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\") pod \"204de64f-65a6-4094-baa1-977345403ba9\" (UID: \"204de64f-65a6-4094-baa1-977345403ba9\") " Oct 30 22:16:45.375475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.375452 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.376095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376059 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376090 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376106 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376119 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204de64f-65a6-4094-baa1-977345403ba9-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376137 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376151 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376165 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376180 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376193 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376207 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376223 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48w4\" (UniqueName: \"kubernetes.io/projected/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-kube-api-access-g48w4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376238 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194dc5f4-757c-422b-a4ee-ac0482d8b0e4-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376251 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.376491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.376308 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.377161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.377140 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "kube-api-access-qcg6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.377232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.377186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.377232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.377194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.377580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.377563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo" pod "204de64f-65a6-4094-baa1-977345403ba9" (UID: "204de64f-65a6-4094-baa1-977345403ba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:45.480192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.479797 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcg6c\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-kube-api-access-qcg6c\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.480192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.479841 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204de64f-65a6-4094-baa1-977345403ba9-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.480192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.479855 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204de64f-65a6-4094-baa1-977345403ba9-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.480192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.479868 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204de64f-65a6-4094-baa1-977345403ba9-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.592544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.592519 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 22:16:45.682543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682501 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.682758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682588 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.682758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682711 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.682758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682750 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.682967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.682967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.683081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682953 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:45.683081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.682970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.683081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683026 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.683279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683218 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\") pod \"4e773594-23d9-405b-80ab-e55075d3a52e\" (UID: \"4e773594-23d9-405b-80ab-e55075d3a52e\") " Oct 30 22:16:45.683279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683245 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.683408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683280 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.683476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.683694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.683626 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.684130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.684041 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.684130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.684073 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.684130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.684090 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.684130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.684106 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e773594-23d9-405b-80ab-e55075d3a52e-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.686133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.686106 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.686308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.686249 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "kube-api-access-crbjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.686796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.686769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.687446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.687421 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo" pod "4e773594-23d9-405b-80ab-e55075d3a52e" (UID: "4e773594-23d9-405b-80ab-e55075d3a52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:45.715210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.715184 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:16:45.786400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.786243 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e773594-23d9-405b-80ab-e55075d3a52e-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.786400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.786295 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.786400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.786316 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbjn\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-kube-api-access-crbjn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.786400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.786332 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e773594-23d9-405b-80ab-e55075d3a52e-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.786400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.786346 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e773594-23d9-405b-80ab-e55075d3a52e-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.844071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.844035 5036 generic.go:356] "Generic (PLEG): container finished" podID="204de64f-65a6-4094-baa1-977345403ba9" containerID="11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301" exitCode=0 Oct 30 22:16:45.844400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.844382 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" Oct 30 22:16:45.846199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.845753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" event={"ID":"204de64f-65a6-4094-baa1-977345403ba9","Type":"ContainerDied","Data":"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301"} Oct 30 22:16:45.846199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.845817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj" event={"ID":"204de64f-65a6-4094-baa1-977345403ba9","Type":"ContainerDied","Data":"de6cebe147de016dbf311c818d4b9725948a3f83fe0c8ce40d6eda6b0f1ae79f"} Oct 30 22:16:45.846199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.845849 5036 scope.go:122] "RemoveContainer" containerID="11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301" Oct 30 22:16:45.850375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.848923 5036 generic.go:356] "Generic (PLEG): container finished" podID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerID="b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89" exitCode=0 Oct 30 22:16:45.850375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.849013 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" event={"ID":"194dc5f4-757c-422b-a4ee-ac0482d8b0e4","Type":"ContainerDied","Data":"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89"} Oct 30 22:16:45.850375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.849057 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" event={"ID":"194dc5f4-757c-422b-a4ee-ac0482d8b0e4","Type":"ContainerDied","Data":"4c53a6de80352e618da9bb281e3356b46f42905018f2abe9d8954cf7489348f1"} Oct 30 22:16:45.850375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.849525 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp" Oct 30 22:16:45.857416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.857008 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" event={"ID":"9fb3042b-235e-4733-ab8e-4552b525905d","Type":"ContainerDied","Data":"3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f"} Oct 30 22:16:45.857416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.857058 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3947065830bfeaee6cb913a93739b251c49cee9a92307e9f372a04b4bcefa70f" Oct 30 22:16:45.857416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.857128 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn" Oct 30 22:16:45.859962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.859807 5036 generic.go:356] "Generic (PLEG): container finished" podID="4e773594-23d9-405b-80ab-e55075d3a52e" containerID="4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be" exitCode=0 Oct 30 22:16:45.859962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.859858 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" event={"ID":"4e773594-23d9-405b-80ab-e55075d3a52e","Type":"ContainerDied","Data":"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be"} Oct 30 22:16:45.859962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.859898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" event={"ID":"4e773594-23d9-405b-80ab-e55075d3a52e","Type":"ContainerDied","Data":"6012942d3540dc26b2b2e4e2ed30c812b08afd3c2347f8a3465992944bd8953f"} Oct 30 22:16:45.860135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.859975 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp" Oct 30 22:16:45.866173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.866156 5036 scope.go:122] "RemoveContainer" containerID="11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301" Oct 30 22:16:45.866474 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:45.866452 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301\": container with ID starting with 11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301 not found: ID does not exist" containerID="11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301" Oct 30 22:16:45.866554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.866487 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301"} err="failed to get container status \"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301\": rpc error: code = NotFound desc = could not find container \"11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301\": container with ID starting with 11b9e4edacd8c0e7c3c8cf3507c398a3a6a94120c96fb330f668222983f00301 not found: ID does not exist" Oct 30 22:16:45.866554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.866506 5036 scope.go:122] "RemoveContainer" containerID="b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89" Oct 30 22:16:45.879242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.879223 5036 scope.go:122] "RemoveContainer" containerID="b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89" Oct 30 22:16:45.879734 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:45.879703 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89\": container with ID starting with b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89 not found: ID does not exist" containerID="b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89" Oct 30 22:16:45.879844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.879743 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89"} err="failed to get container status \"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89\": rpc error: code = NotFound desc = could not find container \"b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89\": container with ID starting with b036aee93e391e74f0a213212f347ed2fd2e4cda9df6636903e89700cc284f89 not found: ID does not exist" Oct 30 22:16:45.879844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.879766 5036 scope.go:122] "RemoveContainer" containerID="4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be" Oct 30 22:16:45.899749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.899712 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 22:16:45.902737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.902719 5036 scope.go:122] "RemoveContainer" containerID="4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be" Oct 30 22:16:45.903170 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:45.903145 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be\": container with ID starting with 4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be not found: ID does not exist" containerID="4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be" Oct 30 22:16:45.903237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.903183 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be"} err="failed to get container status \"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be\": rpc error: code = NotFound desc = could not find container \"4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be\": container with ID starting with 4bfcedd2517540e0bcbeccad34c80ebfa4b2ecbc0a01851b2e3ac50fbc33b3be not found: ID does not exist" Oct 30 22:16:45.905376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.905356 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-hnxrj"] Oct 30 22:16:45.934409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.929624 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 22:16:45.943062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.943023 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-jvrhp"] Oct 30 22:16:45.960615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.960582 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 22:16:45.966167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:45.966143 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-xwzvp"] Oct 30 22:16:46.190869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.190792 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" path="/var/lib/kubelet/pods/194dc5f4-757c-422b-a4ee-ac0482d8b0e4/volumes" Oct 30 22:16:46.191366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.191348 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204de64f-65a6-4094-baa1-977345403ba9" path="/var/lib/kubelet/pods/204de64f-65a6-4094-baa1-977345403ba9/volumes" Oct 30 22:16:46.191841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.191826 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" path="/var/lib/kubelet/pods/4e773594-23d9-405b-80ab-e55075d3a52e/volumes" Oct 30 22:16:46.804080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.804046 5036 scope.go:122] "RemoveContainer" containerID="cc10714da1657fa03c0e5fde962018673604d12b7030391a79d01284b0f9cf77" Oct 30 22:16:46.817293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.817250 5036 scope.go:122] "RemoveContainer" containerID="872996e363dae51c9637857e0f07f60fd740f9902a0acaa1bad0deedbd21b422" Oct 30 22:16:46.830925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.830900 5036 scope.go:122] "RemoveContainer" containerID="65e5319647726ec1c6bf43828c3bb607f34a35f8966026a10b6fd030d49bd702" Oct 30 22:16:46.841986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.841957 5036 scope.go:122] "RemoveContainer" containerID="3c00623eada62a5dafa6a89438848d6b1bf759960ad86e3010767a6b80ce05cd" Oct 30 22:16:46.852813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.852639 5036 scope.go:122] "RemoveContainer" containerID="a1a20b3e9f2cd6ee0cae2229b875ccc4ee8822d934e111f8b5288f6fcb0f7896" Oct 30 22:16:46.864249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:46.864229 5036 scope.go:122] "RemoveContainer" containerID="5966ad500e7bd42f28ca6515fa2449b755b4d3e5eda29da8e69946c0ab0d3596" Oct 30 22:16:47.252701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.252672 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 22:16:47.316116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316084 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316155 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316197 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316374 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316546 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:47.316573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316557 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316716 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.316933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.316781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.317054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.317028 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:47.317125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.317104 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:47.317370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.317342 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:47.317463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.317427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\") pod \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\" (UID: \"47182a02-578a-4d7b-85f9-5bd0352a0f9e\") " Oct 30 22:16:47.317972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.317945 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:47.320245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320205 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:47.320507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320474 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:47.320890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:47.320981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320932 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.320981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320955 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.320981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320972 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.321150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.320986 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.321150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.321002 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.321150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.321017 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47182a02-578a-4d7b-85f9-5bd0352a0f9e-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.321150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.321032 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.321150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.321046 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km" pod "47182a02-578a-4d7b-85f9-5bd0352a0f9e" (UID: "47182a02-578a-4d7b-85f9-5bd0352a0f9e"). InnerVolumeSpecName "kube-api-access-nl4km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:47.426294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.424354 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.426294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.424393 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl4km\" (UniqueName: \"kubernetes.io/projected/47182a02-578a-4d7b-85f9-5bd0352a0f9e-kube-api-access-nl4km\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:47.889611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.889574 5036 generic.go:356] "Generic (PLEG): container finished" podID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerID="60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a" exitCode=0 Oct 30 22:16:47.889785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.889628 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" Oct 30 22:16:47.889785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.889624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" event={"ID":"47182a02-578a-4d7b-85f9-5bd0352a0f9e","Type":"ContainerDied","Data":"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a"} Oct 30 22:16:47.889785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.889695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g" event={"ID":"47182a02-578a-4d7b-85f9-5bd0352a0f9e","Type":"ContainerDied","Data":"b446fddc6a656990c455f041e339fa69aabe3cfffd1306d6ac5ff668de85fd9a"} Oct 30 22:16:47.889785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.889731 5036 scope.go:122] "RemoveContainer" containerID="60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a" Oct 30 22:16:47.906310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.906285 5036 scope.go:122] "RemoveContainer" containerID="60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a" Oct 30 22:16:47.906632 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:47.906612 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a\": container with ID starting with 60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a not found: ID does not exist" containerID="60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a" Oct 30 22:16:47.906708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.906655 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a"} err="failed to get container status \"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a\": rpc error: code = NotFound desc = could not find container \"60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a\": container with ID starting with 60bfb2d70686ec8fd49bc0436d583988955a783987e703bb970c957111a3ac1a not found: ID does not exist" Oct 30 22:16:47.920950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.920928 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 22:16:47.926215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:47.926192 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4j2g"] Oct 30 22:16:48.101479 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:16:48.101442 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:16:48.196651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:48.194208 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" path="/var/lib/kubelet/pods/47182a02-578a-4d7b-85f9-5bd0352a0f9e/volumes" Oct 30 22:16:49.966141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.966106 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:16:49.967052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967033 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" Oct 30 22:16:49.967102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967053 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" Oct 30 22:16:49.967102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967070 5036 cpu_manager.go:414] "containerMap: removing container" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" Oct 30 22:16:49.967102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967077 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967103 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9fb3042b-235e-4733-ab8e-4552b525905d" containerName="revoke-keys" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967109 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb3042b-235e-4733-ab8e-4552b525905d" containerName="revoke-keys" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967119 5036 cpu_manager.go:414] "containerMap: removing container" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967124 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967134 5036 cpu_manager.go:414] "containerMap: removing container" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" Oct 30 22:16:49.967198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967139 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967255 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e773594-23d9-405b-80ab-e55075d3a52e" containerName="istio-proxy" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967281 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="204de64f-65a6-4094-baa1-977345403ba9" containerName="istio-proxy" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967289 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d17b81-8f70-432d-81a0-4b18db51fd3b" containerName="istio-proxy" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967298 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb3042b-235e-4733-ab8e-4552b525905d" containerName="revoke-keys" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967306 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="47182a02-578a-4d7b-85f9-5bd0352a0f9e" containerName="istio-proxy" Oct 30 22:16:49.967404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.967313 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="194dc5f4-757c-422b-a4ee-ac0482d8b0e4" containerName="istio-proxy" Oct 30 22:16:49.972796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.972776 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:49.994041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.989331 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:16:49.994882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.994863 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:49.999100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:49.998171 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:16:50.015558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.015527 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:16:50.081730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.081694 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.081935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.081746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.081935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.081851 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.081959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.081979 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082115 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082143 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srcch\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.082332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082322 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082364 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082398 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082445 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbn5c\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082475 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.082675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.082496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.183874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.183838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.183874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.183885 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184117 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.184227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184207 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184332 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184621 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184674 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srcch\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184744 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.184808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184723 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.184808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184747 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.184958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184826 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.184958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184856 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.184958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184904 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184976 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbn5c\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.184985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185011 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.185105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185039 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185073 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.185409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185215 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.185409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.185409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185302 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185514 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185670 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.185923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.185902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.188929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.188902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.189208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.189181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.189496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.189468 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.190020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.189986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.197145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.197121 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbn5c\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.197590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.197566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.198112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.198088 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.198950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.198923 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srcch\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.293306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.293198 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:50.318224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.318199 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:50.605989 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:50.605953 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d298df_47a0_4a4c_a233_680e26b293df.slice/crio-23fae5d3ba0391b0c2e61940c149eadc2ca3927d06e0eb75b8fad73be3f3ba30 WatchSource:0}: Error finding container 23fae5d3ba0391b0c2e61940c149eadc2ca3927d06e0eb75b8fad73be3f3ba30: Status 404 returned error can't find the container with id 23fae5d3ba0391b0c2e61940c149eadc2ca3927d06e0eb75b8fad73be3f3ba30 Oct 30 22:16:50.606147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.606130 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:16:50.609357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.609309 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.609483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.609447 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.609562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.609537 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.648073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.647696 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:16:50.648874 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:50.648850 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ba631b_dc79_4aa7_97c9_9638d3037991.slice/crio-1241eef8210801b79610cd4da3483fbdc9ab88230070613d74146a424754bd31 WatchSource:0}: Error finding container 1241eef8210801b79610cd4da3483fbdc9ab88230070613d74146a424754bd31: Status 404 returned error can't find the container with id 1241eef8210801b79610cd4da3483fbdc9ab88230070613d74146a424754bd31 Oct 30 22:16:50.652405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.652369 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.652504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.652438 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.652504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.652479 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:50.922718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.922390 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" event={"ID":"f4d298df-47a0-4a4c-a233-680e26b293df","Type":"ContainerStarted","Data":"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f"} Oct 30 22:16:50.922718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.922459 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" event={"ID":"f4d298df-47a0-4a4c-a233-680e26b293df","Type":"ContainerStarted","Data":"23fae5d3ba0391b0c2e61940c149eadc2ca3927d06e0eb75b8fad73be3f3ba30"} Oct 30 22:16:50.925489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.925463 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" event={"ID":"e9ba631b-dc79-4aa7-97c9-9638d3037991","Type":"ContainerStarted","Data":"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04"} Oct 30 22:16:50.926571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.926546 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" event={"ID":"e9ba631b-dc79-4aa7-97c9-9638d3037991","Type":"ContainerStarted","Data":"1241eef8210801b79610cd4da3483fbdc9ab88230070613d74146a424754bd31"} Oct 30 22:16:50.948290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.945905 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podStartSLOduration=1.945886068 podStartE2EDuration="1.945886068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.944160243 +0000 UTC m=+2234.894115194" watchObservedRunningTime="2026-10-30 22:16:50.945886068 +0000 UTC m=+2234.895841014" Oct 30 22:16:50.972799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:50.972734 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podStartSLOduration=1.972715512 podStartE2EDuration="1.972715512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.967594453 +0000 UTC m=+2234.917549399" watchObservedRunningTime="2026-10-30 22:16:50.972715512 +0000 UTC m=+2234.922670459" Oct 30 22:16:51.295754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.294707 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:51.296937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.296901 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.297191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.297162 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:51.330375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.330338 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:51.331727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.331694 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.331932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:51.331908 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:52.294474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.294431 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.294942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.294490 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:52.329291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.324723 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.329291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.324780 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:52.934364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.934326 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9"] Oct 30 22:16:52.969563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.969534 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9"] Oct 30 22:16:52.969901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.969879 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:52.977345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.976127 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:52.977345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.976685 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:52.977345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.976764 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:16:52.977345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:52.976830 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:53.004027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.003993 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdlr\" (UniqueName: \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.004193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.004125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.004656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.004630 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.106068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.106039 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdlr\" (UniqueName: \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.106307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.106123 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.106307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.106219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.107115 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.107053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.109023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.109005 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.139926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.129909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdlr\" (UniqueName: \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\") pod \"maas-api-revoke-sub-c996765f-qxlr9\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.294499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.294418 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.294499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.294475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:53.298763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.298741 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:53.329300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.321691 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.329300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.321750 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:53.886460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:53.886028 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9"] Oct 30 22:16:53.892071 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:16:53.891767 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51108ee9_c2a3_4059_9329_9a3f770d9b22.slice/crio-b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b WatchSource:0}: Error finding container b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b: Status 404 returned error can't find the container with id b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b Oct 30 22:16:54.062301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:54.061644 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" event={"ID":"51108ee9-c2a3-4059-9329-9a3f770d9b22","Type":"ContainerStarted","Data":"b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b"} Oct 30 22:16:54.294298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:54.294234 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.294501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:54.294306 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:54.324302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:54.318652 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.324302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:54.318722 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:55.087751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.087695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" event={"ID":"51108ee9-c2a3-4059-9329-9a3f770d9b22","Type":"ContainerStarted","Data":"ba7f4b8aa7f66b09141793be392100df6247d268dd10d4f106bd2c4884ee9ebb"} Oct 30 22:16:55.109111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.109065 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" podStartSLOduration=3.109051645 podStartE2EDuration="3.109051645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:55.105225927 +0000 UTC m=+2239.055180872" watchObservedRunningTime="2026-10-30 22:16:55.109051645 +0000 UTC m=+2239.059006590" Oct 30 22:16:55.293938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.293906 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.294103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.293957 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:55.318936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.318561 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.318936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:55.318619 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:56.100043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.100013 5036 generic.go:356] "Generic (PLEG): container finished" podID="51108ee9-c2a3-4059-9329-9a3f770d9b22" containerID="ba7f4b8aa7f66b09141793be392100df6247d268dd10d4f106bd2c4884ee9ebb" exitCode=0 Oct 30 22:16:56.100475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.100083 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" event={"ID":"51108ee9-c2a3-4059-9329-9a3f770d9b22","Type":"ContainerDied","Data":"ba7f4b8aa7f66b09141793be392100df6247d268dd10d4f106bd2c4884ee9ebb"} Oct 30 22:16:56.293758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.293717 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 30 22:16:56.293936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.293794 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 30 22:16:56.322684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.322646 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 30 22:16:56.323362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:56.322971 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 30 22:16:57.479427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.478984 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:57.484627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.481326 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:57.484627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.483366 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:16:57.494330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.492984 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:57.494330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.494301 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:57.497909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.497039 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:16:57.499255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.499166 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:57.642962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.640224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\") pod \"51108ee9-c2a3-4059-9329-9a3f770d9b22\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " Oct 30 22:16:57.642962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.641486 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\" (UniqueName: \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\") pod \"51108ee9-c2a3-4059-9329-9a3f770d9b22\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " Oct 30 22:16:57.642962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.641805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\") pod \"51108ee9-c2a3-4059-9329-9a3f770d9b22\" (UID: \"51108ee9-c2a3-4059-9329-9a3f770d9b22\") " Oct 30 22:16:57.644031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.643890 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca" pod "51108ee9-c2a3-4059-9329-9a3f770d9b22" (UID: "51108ee9-c2a3-4059-9329-9a3f770d9b22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:57.647140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.647098 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr" pod "51108ee9-c2a3-4059-9329-9a3f770d9b22" (UID: "51108ee9-c2a3-4059-9329-9a3f770d9b22"). InnerVolumeSpecName "kube-api-access-lxdlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:57.651757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.651735 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca" pod "51108ee9-c2a3-4059-9329-9a3f770d9b22" (UID: "51108ee9-c2a3-4059-9329-9a3f770d9b22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:57.744135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.744099 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.744135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.744130 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51108ee9-c2a3-4059-9329-9a3f770d9b22-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.744135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:57.744143 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdlr\" (UniqueName: \"kubernetes.io/projected/51108ee9-c2a3-4059-9329-9a3f770d9b22-kube-api-access-lxdlr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:58.123887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:58.123799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" Oct 30 22:16:58.123887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:58.123835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9" event={"ID":"51108ee9-c2a3-4059-9329-9a3f770d9b22","Type":"ContainerDied","Data":"b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b"} Oct 30 22:16:58.124130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:16:58.123898 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ace144ada11dcc4719a568ab8fc318d515b6351508deb8d4693b5b70ef086b" Oct 30 22:17:03.508895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.508857 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt"] Oct 30 22:17:03.509813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.509793 5036 cpu_manager.go:414] "containerMap: removing container" podUID="51108ee9-c2a3-4059-9329-9a3f770d9b22" containerName="revoke-keys" Oct 30 22:17:03.509869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.509815 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51108ee9-c2a3-4059-9329-9a3f770d9b22" containerName="revoke-keys" Oct 30 22:17:03.509951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.509940 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="51108ee9-c2a3-4059-9329-9a3f770d9b22" containerName="revoke-keys" Oct 30 22:17:03.514248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.514230 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.517768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.517740 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:03.521427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.518374 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:17:03.521427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.518481 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:03.521427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.519815 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:03.522889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.522865 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt"] Oct 30 22:17:03.596920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.596885 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.597075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.596934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.597213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.597191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm85z\" (UniqueName: \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.720303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.719575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm85z\" (UniqueName: \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.720303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.719806 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.720303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.719882 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.724667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.722258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.729340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.729310 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.762005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.757632 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm85z\" (UniqueName: \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\") pod \"maas-api-revoke-sub-2945cc1e-4msvt\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:03.834073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:03.834038 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:04.132244 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:04.132210 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8db2bce_0087_4340_918f_d9e43bc8194e.slice/crio-dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c WatchSource:0}: Error finding container dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c: Status 404 returned error can't find the container with id dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c Oct 30 22:17:04.132481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:04.132457 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt"] Oct 30 22:17:04.192614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:04.192583 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" event={"ID":"e8db2bce-0087-4340-918f-d9e43bc8194e","Type":"ContainerStarted","Data":"dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c"} Oct 30 22:17:05.206161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:05.206125 5036 generic.go:356] "Generic (PLEG): container finished" podID="e8db2bce-0087-4340-918f-d9e43bc8194e" containerID="868b46f52c4dc969a812aa6605e49367b47662ae80b0b0554e90d6bfa15a6aa5" exitCode=0 Oct 30 22:17:05.206661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:05.206317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" event={"ID":"e8db2bce-0087-4340-918f-d9e43bc8194e","Type":"ContainerDied","Data":"868b46f52c4dc969a812aa6605e49367b47662ae80b0b0554e90d6bfa15a6aa5"} Oct 30 22:17:06.403173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.403140 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:06.474598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.474508 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\") pod \"e8db2bce-0087-4340-918f-d9e43bc8194e\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " Oct 30 22:17:06.474749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.474615 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\") pod \"e8db2bce-0087-4340-918f-d9e43bc8194e\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " Oct 30 22:17:06.475044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.475011 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca" pod "e8db2bce-0087-4340-918f-d9e43bc8194e" (UID: "e8db2bce-0087-4340-918f-d9e43bc8194e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:06.475234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.475214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\" (UniqueName: \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\") pod \"e8db2bce-0087-4340-918f-d9e43bc8194e\" (UID: \"e8db2bce-0087-4340-918f-d9e43bc8194e\") " Oct 30 22:17:06.476113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.476087 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:06.490355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.477702 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca" pod "e8db2bce-0087-4340-918f-d9e43bc8194e" (UID: "e8db2bce-0087-4340-918f-d9e43bc8194e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:06.490355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.477986 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z" pod "e8db2bce-0087-4340-918f-d9e43bc8194e" (UID: "e8db2bce-0087-4340-918f-d9e43bc8194e"). InnerVolumeSpecName "kube-api-access-nm85z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:06.577929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.577886 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8db2bce-0087-4340-918f-d9e43bc8194e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:06.577929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:06.577925 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm85z\" (UniqueName: \"kubernetes.io/projected/e8db2bce-0087-4340-918f-d9e43bc8194e-kube-api-access-nm85z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:07.227227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:07.227189 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" Oct 30 22:17:07.227227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:07.227215 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt" event={"ID":"e8db2bce-0087-4340-918f-d9e43bc8194e","Type":"ContainerDied","Data":"dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c"} Oct 30 22:17:07.227515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:07.227290 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf8b960d4bc46dfb278764d7d8fa044c8284b2b35a9adbace61651bb8f4715c" Oct 30 22:17:16.703348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.703303 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7"] Oct 30 22:17:16.704220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.704203 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e8db2bce-0087-4340-918f-d9e43bc8194e" containerName="revoke-keys" Oct 30 22:17:16.704220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.704221 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8db2bce-0087-4340-918f-d9e43bc8194e" containerName="revoke-keys" Oct 30 22:17:16.704394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.704383 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8db2bce-0087-4340-918f-d9e43bc8194e" containerName="revoke-keys" Oct 30 22:17:16.709629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.709609 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.717766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.717192 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:16.717911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.717875 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:16.717976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.717955 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:16.718114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.718058 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:17:16.718884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.718863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7"] Oct 30 22:17:16.794851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.794819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgzl\" (UniqueName: \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.795039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.794973 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.795124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.795109 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.896889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.896853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgzl\" (UniqueName: \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.897109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.896985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.897203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.897107 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.897934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.897904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.901374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.901347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:16.909217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:16.909185 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgzl\" (UniqueName: \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\") pod \"maas-api-revoke-sub-ee52cd56-6bpc7\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:17.050363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:17.050276 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:17.405941 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:17.405905 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e424c15_66c3_42e3_b92e_729a99a87876.slice/crio-e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391 WatchSource:0}: Error finding container e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391: Status 404 returned error can't find the container with id e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391 Oct 30 22:17:17.406438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:17.406366 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7"] Oct 30 22:17:18.101087 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:18.101052 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:17:18.501555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:18.501305 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" event={"ID":"0e424c15-66c3-42e3-b92e-729a99a87876","Type":"ContainerStarted","Data":"81a76cab1b841e3fc9881df34cb79f74dd2b936113733b7e311fd1671e2b8bf7"} Oct 30 22:17:18.501555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:18.501381 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" event={"ID":"0e424c15-66c3-42e3-b92e-729a99a87876","Type":"ContainerStarted","Data":"e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391"} Oct 30 22:17:18.520303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:18.519960 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" podStartSLOduration=2.5199422670000002 podStartE2EDuration="2.519942267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:18.51885574 +0000 UTC m=+2262.468810685" watchObservedRunningTime="2026-10-30 22:17:18.519942267 +0000 UTC m=+2262.469897215" Oct 30 22:17:19.623768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:19.623722 5036 generic.go:356] "Generic (PLEG): container finished" podID="0e424c15-66c3-42e3-b92e-729a99a87876" containerID="81a76cab1b841e3fc9881df34cb79f74dd2b936113733b7e311fd1671e2b8bf7" exitCode=0 Oct 30 22:17:19.624318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:19.623884 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" event={"ID":"0e424c15-66c3-42e3-b92e-729a99a87876","Type":"ContainerDied","Data":"81a76cab1b841e3fc9881df34cb79f74dd2b936113733b7e311fd1671e2b8bf7"} Oct 30 22:17:20.905157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:20.905084 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:21.067081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.067046 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\" (UniqueName: \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\") pod \"0e424c15-66c3-42e3-b92e-729a99a87876\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " Oct 30 22:17:21.067325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.067306 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\") pod \"0e424c15-66c3-42e3-b92e-729a99a87876\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " Oct 30 22:17:21.067454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.067437 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\") pod \"0e424c15-66c3-42e3-b92e-729a99a87876\" (UID: \"0e424c15-66c3-42e3-b92e-729a99a87876\") " Oct 30 22:17:21.067618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.067595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca" pod "0e424c15-66c3-42e3-b92e-729a99a87876" (UID: "0e424c15-66c3-42e3-b92e-729a99a87876"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:21.068318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.068203 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:21.069692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.069670 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl" pod "0e424c15-66c3-42e3-b92e-729a99a87876" (UID: "0e424c15-66c3-42e3-b92e-729a99a87876"). InnerVolumeSpecName "kube-api-access-9rgzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:21.070036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.070014 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca" pod "0e424c15-66c3-42e3-b92e-729a99a87876" (UID: "0e424c15-66c3-42e3-b92e-729a99a87876"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:21.170441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.170350 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e424c15-66c3-42e3-b92e-729a99a87876-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:21.170441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.170382 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgzl\" (UniqueName: \"kubernetes.io/projected/0e424c15-66c3-42e3-b92e-729a99a87876-kube-api-access-9rgzl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:21.649219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.649184 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" Oct 30 22:17:21.649219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.649210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7" event={"ID":"0e424c15-66c3-42e3-b92e-729a99a87876","Type":"ContainerDied","Data":"e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391"} Oct 30 22:17:21.649483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:21.649281 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0275f029bd1f23ea339d9db002ffb60fcaa1e0ad91b69d8b913cb73fda32391" Oct 30 22:17:23.282590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.282554 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b"] Oct 30 22:17:23.283418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.283401 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0e424c15-66c3-42e3-b92e-729a99a87876" containerName="revoke-keys" Oct 30 22:17:23.283486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.283419 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e424c15-66c3-42e3-b92e-729a99a87876" containerName="revoke-keys" Oct 30 22:17:23.283569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.283559 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e424c15-66c3-42e3-b92e-729a99a87876" containerName="revoke-keys" Oct 30 22:17:23.287739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.287719 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.297013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.296986 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:23.297167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.297109 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:23.297167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.296986 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:23.297601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.297567 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:17:23.299129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.299106 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b"] Oct 30 22:17:23.426102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.426065 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.426353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.426168 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmzm\" (UniqueName: \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.426522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.426500 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.528145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.528109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.528394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.528192 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmzm\" (UniqueName: \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.528487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.528461 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.529124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.529099 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.530976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.530950 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.540433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.540370 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmzm\" (UniqueName: \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\") pod \"maas-api-revoke-sub-67c739f8-t8z6b\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.607803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.607773 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:23.924098 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:23.924058 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1afbc47_0c4f_4b55_9053_d5dd35541e74.slice/crio-222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483 WatchSource:0}: Error finding container 222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483: Status 404 returned error can't find the container with id 222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483 Oct 30 22:17:23.924394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:23.924363 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b"] Oct 30 22:17:24.066413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.066380 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx"] Oct 30 22:17:24.072808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.072782 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.079172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.079144 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx"] Oct 30 22:17:24.248942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.248855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pld\" (UniqueName: \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.249187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.249158 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.249426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.249404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.351175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.351143 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.351600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.351315 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pld\" (UniqueName: \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.351677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.351614 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.351895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.351871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.354311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.354244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.363528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.363501 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pld\" (UniqueName: \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\") pod \"maas-api-revoke-sub-fc2f674a-jh8lx\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.392475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.392453 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:24.680505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.680469 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" event={"ID":"c1afbc47-0c4f-4b55-9053-d5dd35541e74","Type":"ContainerStarted","Data":"95b5c8d2da6bee854dfd7f6b56299a443cfcb2919b8378acfcbefbf0f9b25e74"} Oct 30 22:17:24.680730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.680532 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" event={"ID":"c1afbc47-0c4f-4b55-9053-d5dd35541e74","Type":"ContainerStarted","Data":"222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483"} Oct 30 22:17:24.689233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.689200 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx"] Oct 30 22:17:24.689392 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:24.689374 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9942370d_4276_4607_83c7_6fc1d20a1a9e.slice/crio-410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516 WatchSource:0}: Error finding container 410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516: Status 404 returned error can't find the container with id 410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516 Oct 30 22:17:24.701130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:24.701076 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" podStartSLOduration=1.701059553 podStartE2EDuration="1.701059553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.697673721 +0000 UTC m=+2268.647628692" watchObservedRunningTime="2026-10-30 22:17:24.701059553 +0000 UTC m=+2268.651014500" Oct 30 22:17:25.044357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.044328 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz"] Oct 30 22:17:25.050354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.050316 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.060203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.060172 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz"] Oct 30 22:17:25.175737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.175706 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.175908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.175864 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.176021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.175999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htccp\" (UniqueName: \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.278252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.278214 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.278442 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.278418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.278559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.278490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htccp\" (UniqueName: \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.279120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.279091 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.281309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.281259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.290514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.290492 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htccp\" (UniqueName: \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\") pod \"maas-api-revoke-sub-a30772e6-fqrkz\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.403281 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.403221 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:25.691895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.691844 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" event={"ID":"9942370d-4276-4607-83c7-6fc1d20a1a9e","Type":"ContainerStarted","Data":"28a018f7e741015bb3d67541d4cc0a8e772b01de29452a932c11f9dcfc4ae1b6"} Oct 30 22:17:25.692086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.691921 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" event={"ID":"9942370d-4276-4607-83c7-6fc1d20a1a9e","Type":"ContainerStarted","Data":"410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516"} Oct 30 22:17:25.694021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.693992 5036 generic.go:356] "Generic (PLEG): container finished" podID="c1afbc47-0c4f-4b55-9053-d5dd35541e74" containerID="95b5c8d2da6bee854dfd7f6b56299a443cfcb2919b8378acfcbefbf0f9b25e74" exitCode=0 Oct 30 22:17:25.694168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.694081 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" event={"ID":"c1afbc47-0c4f-4b55-9053-d5dd35541e74","Type":"ContainerDied","Data":"95b5c8d2da6bee854dfd7f6b56299a443cfcb2919b8378acfcbefbf0f9b25e74"} Oct 30 22:17:25.710730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.710676 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" podStartSLOduration=1.710658337 podStartE2EDuration="1.710658337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:25.709578122 +0000 UTC m=+2269.659533082" watchObservedRunningTime="2026-10-30 22:17:25.710658337 +0000 UTC m=+2269.660613283" Oct 30 22:17:25.744474 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:25.744434 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a075d7_a1f9_4d51_bd30_51fb3e63a71c.slice/crio-796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54 WatchSource:0}: Error finding container 796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54: Status 404 returned error can't find the container with id 796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54 Oct 30 22:17:25.744932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:25.744907 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz"] Oct 30 22:17:26.764538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:26.764487 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" event={"ID":"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c","Type":"ContainerStarted","Data":"c23dcf5b60cfde1e702135a0ecd7ad40a6253f16e3e5684fef06dab2ecbfdc5a"} Oct 30 22:17:26.765121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:26.764615 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" event={"ID":"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c","Type":"ContainerStarted","Data":"796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54"} Oct 30 22:17:26.782808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:26.778105 5036 generic.go:356] "Generic (PLEG): container finished" podID="9942370d-4276-4607-83c7-6fc1d20a1a9e" containerID="28a018f7e741015bb3d67541d4cc0a8e772b01de29452a932c11f9dcfc4ae1b6" exitCode=0 Oct 30 22:17:26.782808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:26.778676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" event={"ID":"9942370d-4276-4607-83c7-6fc1d20a1a9e","Type":"ContainerDied","Data":"28a018f7e741015bb3d67541d4cc0a8e772b01de29452a932c11f9dcfc4ae1b6"} Oct 30 22:17:26.793624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:26.793562 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" podStartSLOduration=1.793542805 podStartE2EDuration="1.793542805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:26.782182658 +0000 UTC m=+2270.732137612" watchObservedRunningTime="2026-10-30 22:17:26.793542805 +0000 UTC m=+2270.743497752" Oct 30 22:17:27.007897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.007874 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:27.087581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.087495 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\" (UniqueName: \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\") pod \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " Oct 30 22:17:27.087747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.087592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\") pod \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " Oct 30 22:17:27.087962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.087940 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\") pod \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\" (UID: \"c1afbc47-0c4f-4b55-9053-d5dd35541e74\") " Oct 30 22:17:27.088401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.088354 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca" pod "c1afbc47-0c4f-4b55-9053-d5dd35541e74" (UID: "c1afbc47-0c4f-4b55-9053-d5dd35541e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:27.088764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.088731 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.092783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.090257 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm" pod "c1afbc47-0c4f-4b55-9053-d5dd35541e74" (UID: "c1afbc47-0c4f-4b55-9053-d5dd35541e74"). InnerVolumeSpecName "kube-api-access-whmzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:27.092783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.091127 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca" pod "c1afbc47-0c4f-4b55-9053-d5dd35541e74" (UID: "c1afbc47-0c4f-4b55-9053-d5dd35541e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:27.189915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.189883 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmzm\" (UniqueName: \"kubernetes.io/projected/c1afbc47-0c4f-4b55-9053-d5dd35541e74-kube-api-access-whmzm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.189915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.189912 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1afbc47-0c4f-4b55-9053-d5dd35541e74-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.792055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.791813 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:17:27.803037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.803008 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" Oct 30 22:17:27.804157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.803208 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b" event={"ID":"c1afbc47-0c4f-4b55-9053-d5dd35541e74","Type":"ContainerDied","Data":"222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483"} Oct 30 22:17:27.819183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.809724 5036 generic.go:356] "Generic (PLEG): container finished" podID="c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" containerID="c23dcf5b60cfde1e702135a0ecd7ad40a6253f16e3e5684fef06dab2ecbfdc5a" exitCode=0 Oct 30 22:17:27.846376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.846346 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="222651b74e21381a46354d887196dcf942f26603555e68da16d90769b06a2483" Oct 30 22:17:27.846553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:27.846379 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" event={"ID":"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c","Type":"ContainerDied","Data":"c23dcf5b60cfde1e702135a0ecd7ad40a6253f16e3e5684fef06dab2ecbfdc5a"} Oct 30 22:17:28.516015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.515990 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:28.614951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.614913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\" (UniqueName: \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\") pod \"9942370d-4276-4607-83c7-6fc1d20a1a9e\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " Oct 30 22:17:28.615185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.614995 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\") pod \"9942370d-4276-4607-83c7-6fc1d20a1a9e\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " Oct 30 22:17:28.615185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.615111 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\") pod \"9942370d-4276-4607-83c7-6fc1d20a1a9e\" (UID: \"9942370d-4276-4607-83c7-6fc1d20a1a9e\") " Oct 30 22:17:28.615476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.615446 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca" pod "9942370d-4276-4607-83c7-6fc1d20a1a9e" (UID: "9942370d-4276-4607-83c7-6fc1d20a1a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:28.615896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.615869 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.617525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.617501 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld" pod "9942370d-4276-4607-83c7-6fc1d20a1a9e" (UID: "9942370d-4276-4607-83c7-6fc1d20a1a9e"). InnerVolumeSpecName "kube-api-access-m6pld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:28.617632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.617606 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca" pod "9942370d-4276-4607-83c7-6fc1d20a1a9e" (UID: "9942370d-4276-4607-83c7-6fc1d20a1a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:28.717862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.717827 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pld\" (UniqueName: \"kubernetes.io/projected/9942370d-4276-4607-83c7-6fc1d20a1a9e-kube-api-access-m6pld\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.717862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.717857 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9942370d-4276-4607-83c7-6fc1d20a1a9e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.821477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.821384 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" Oct 30 22:17:28.821477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.821390 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx" event={"ID":"9942370d-4276-4607-83c7-6fc1d20a1a9e","Type":"ContainerDied","Data":"410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516"} Oct 30 22:17:28.821477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:28.821454 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410a53678dcdc682f1f7931e0220393ce623ffc9c6c247647ffc03100c84a516" Oct 30 22:17:29.093824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.093801 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:29.125679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.123648 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\") pod \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " Oct 30 22:17:29.125679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.123717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\") pod \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " Oct 30 22:17:29.125679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.123899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\" (UniqueName: \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\") pod \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\" (UID: \"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c\") " Oct 30 22:17:29.125679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.125340 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca" pod "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" (UID: "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:29.127423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.127395 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca" pod "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" (UID: "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:29.128238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.128208 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp" pod "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" (UID: "c3a075d7-a1f9-4d51-bd30-51fb3e63a71c"). InnerVolumeSpecName "kube-api-access-htccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:29.226226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.226191 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:29.226226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.226236 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:29.226520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.226256 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htccp\" (UniqueName: \"kubernetes.io/projected/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c-kube-api-access-htccp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:29.834253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.834157 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" event={"ID":"c3a075d7-a1f9-4d51-bd30-51fb3e63a71c","Type":"ContainerDied","Data":"796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54"} Oct 30 22:17:29.834253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.834220 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796401fbefa67580f6666b3c979ebfe45e8a5a72147bd4977257898e4893de54" Oct 30 22:17:29.834801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:29.834307 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz" Oct 30 22:17:31.342214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.342174 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c"] Oct 30 22:17:31.343051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343033 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9942370d-4276-4607-83c7-6fc1d20a1a9e" containerName="revoke-keys" Oct 30 22:17:31.343051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343051 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9942370d-4276-4607-83c7-6fc1d20a1a9e" containerName="revoke-keys" Oct 30 22:17:31.343182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343064 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" containerName="revoke-keys" Oct 30 22:17:31.343182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343070 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" containerName="revoke-keys" Oct 30 22:17:31.343182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343098 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c1afbc47-0c4f-4b55-9053-d5dd35541e74" containerName="revoke-keys" Oct 30 22:17:31.343182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343103 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1afbc47-0c4f-4b55-9053-d5dd35541e74" containerName="revoke-keys" Oct 30 22:17:31.343489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343242 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9942370d-4276-4607-83c7-6fc1d20a1a9e" containerName="revoke-keys" Oct 30 22:17:31.343489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343254 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" containerName="revoke-keys" Oct 30 22:17:31.343489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.343282 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1afbc47-0c4f-4b55-9053-d5dd35541e74" containerName="revoke-keys" Oct 30 22:17:31.347496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.347475 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.352240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.351982 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:31.352240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.352056 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:31.352240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.352082 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:31.352834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.352248 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:17:31.358831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.358802 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c"] Oct 30 22:17:31.374537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.374510 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.374700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.374576 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.374759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.374690 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrc5\" (UniqueName: \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.476100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.476061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.476324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.476125 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.476324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.476147 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrc5\" (UniqueName: \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.476860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.476835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.478820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.478801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.488428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.488400 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrc5\" (UniqueName: \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\") pod \"maas-api-revoke-sub-30332d58-7d57c\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.686356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.677453 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:31.991868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:31.991833 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c"] Oct 30 22:17:31.992195 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:31.992168 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89dd5d3_446d_47e4_a65a_8462a4730638.slice/crio-ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416 WatchSource:0}: Error finding container ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416: Status 404 returned error can't find the container with id ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416 Oct 30 22:17:32.792422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:32.792335 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" podUID="b525a859-41f6-4667-b69d-6cfbb54e03e4" containerName="payload-processing" containerID="cri-o://3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457" gracePeriod=25 Oct 30 22:17:32.872420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:32.872378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" event={"ID":"d89dd5d3-446d-47e4-a65a-8462a4730638","Type":"ContainerStarted","Data":"cb5b1abeff7eea4fbfa0f06ca7f9f98c29221493fc2583acb63012a26eae8c19"} Oct 30 22:17:32.872603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:32.872456 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" event={"ID":"d89dd5d3-446d-47e4-a65a-8462a4730638","Type":"ContainerStarted","Data":"ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416"} Oct 30 22:17:32.890403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:32.890354 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" podStartSLOduration=1.890342048 podStartE2EDuration="1.890342048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:32.888742437 +0000 UTC m=+2276.838697382" watchObservedRunningTime="2026-10-30 22:17:32.890342048 +0000 UTC m=+2276.840296993" Oct 30 22:17:33.186336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.186308 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:17:33.211203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.211168 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\") pod \"b525a859-41f6-4667-b69d-6cfbb54e03e4\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " Oct 30 22:17:33.211951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.211835 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume" pod "b525a859-41f6-4667-b69d-6cfbb54e03e4" (UID: "b525a859-41f6-4667-b69d-6cfbb54e03e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:33.212092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.211970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\") pod \"b525a859-41f6-4667-b69d-6cfbb54e03e4\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " Oct 30 22:17:33.212157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.212121 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\") pod \"b525a859-41f6-4667-b69d-6cfbb54e03e4\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " Oct 30 22:17:33.212212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.212188 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\") pod \"b525a859-41f6-4667-b69d-6cfbb54e03e4\" (UID: \"b525a859-41f6-4667-b69d-6cfbb54e03e4\") " Oct 30 22:17:33.212786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.212754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume" pod "b525a859-41f6-4667-b69d-6cfbb54e03e4" (UID: "b525a859-41f6-4667-b69d-6cfbb54e03e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:33.213671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.213645 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:33.213790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.213675 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b525a859-41f6-4667-b69d-6cfbb54e03e4-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:33.214990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.214964 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials" pod "b525a859-41f6-4667-b69d-6cfbb54e03e4" (UID: "b525a859-41f6-4667-b69d-6cfbb54e03e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:33.215670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.215644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm" pod "b525a859-41f6-4667-b69d-6cfbb54e03e4" (UID: "b525a859-41f6-4667-b69d-6cfbb54e03e4"). InnerVolumeSpecName "kube-api-access-ldxqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:33.315404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.315319 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:33.315404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.315357 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxqm\" (UniqueName: \"kubernetes.io/projected/b525a859-41f6-4667-b69d-6cfbb54e03e4-kube-api-access-ldxqm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:33.883494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.883460 5036 generic.go:356] "Generic (PLEG): container finished" podID="d89dd5d3-446d-47e4-a65a-8462a4730638" containerID="cb5b1abeff7eea4fbfa0f06ca7f9f98c29221493fc2583acb63012a26eae8c19" exitCode=0 Oct 30 22:17:33.883933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.883546 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" event={"ID":"d89dd5d3-446d-47e4-a65a-8462a4730638","Type":"ContainerDied","Data":"cb5b1abeff7eea4fbfa0f06ca7f9f98c29221493fc2583acb63012a26eae8c19"} Oct 30 22:17:33.885105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.885084 5036 generic.go:356] "Generic (PLEG): container finished" podID="b525a859-41f6-4667-b69d-6cfbb54e03e4" containerID="3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457" exitCode=0 Oct 30 22:17:33.885187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.885133 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" Oct 30 22:17:33.885187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.885170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" event={"ID":"b525a859-41f6-4667-b69d-6cfbb54e03e4","Type":"ContainerDied","Data":"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457"} Oct 30 22:17:33.885287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.885226 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6" event={"ID":"b525a859-41f6-4667-b69d-6cfbb54e03e4","Type":"ContainerDied","Data":"83df9c167b3cb86b00cd00972a5274fb65a0f8cc70a134bb1a4100bb85e848fa"} Oct 30 22:17:33.885287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.885255 5036 scope.go:122] "RemoveContainer" containerID="3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457" Oct 30 22:17:33.909520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.897546 5036 scope.go:122] "RemoveContainer" containerID="3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457" Oct 30 22:17:33.909520 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:33.897856 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457\": container with ID starting with 3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457 not found: ID does not exist" containerID="3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457" Oct 30 22:17:33.909520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.897890 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457"} err="failed to get container status \"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457\": rpc error: code = NotFound desc = could not find container \"3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457\": container with ID starting with 3d50ecdada72366aaa2bce7440f0690a06f0cc25487624501c92e8727ec3d457 not found: ID does not exist" Oct 30 22:17:33.982781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.981616 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd"] Oct 30 22:17:33.985922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.983604 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b525a859-41f6-4667-b69d-6cfbb54e03e4" containerName="payload-processing" Oct 30 22:17:33.985922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.983628 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b525a859-41f6-4667-b69d-6cfbb54e03e4" containerName="payload-processing" Oct 30 22:17:33.985922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:33.983871 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b525a859-41f6-4667-b69d-6cfbb54e03e4" containerName="payload-processing" Oct 30 22:17:34.007144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.007110 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd"] Oct 30 22:17:34.007342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.007244 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.059708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.059678 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:17:34.066746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.066725 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7cf8z9xg6"] Oct 30 22:17:34.136911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.136814 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcxb\" (UniqueName: \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.136911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.136886 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.137194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.137166 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.194212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.194178 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b525a859-41f6-4667-b69d-6cfbb54e03e4" path="/var/lib/kubelet/pods/b525a859-41f6-4667-b69d-6cfbb54e03e4/volumes" Oct 30 22:17:34.238623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.238587 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.238808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.238774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcxb\" (UniqueName: \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.238893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.238823 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.239386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.239363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.241418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.241392 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.255782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.255750 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcxb\" (UniqueName: \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\") pod \"maas-api-revoke-sub-11073557-kgdhd\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.329279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.329233 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:34.595823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.595792 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt"] Oct 30 22:17:34.602029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.602002 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.612217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.612167 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt"] Oct 30 22:17:34.627425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.627380 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd"] Oct 30 22:17:34.630757 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:34.630586 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4228e118_12fe_4495_83de_10dd93af169b.slice/crio-24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0 WatchSource:0}: Error finding container 24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0: Status 404 returned error can't find the container with id 24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0 Oct 30 22:17:34.646145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.646093 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.646411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.646387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dv2\" (UniqueName: \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.646846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.646800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.751114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.749222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.751114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.749379 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.751114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.749488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dv2\" (UniqueName: \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.751114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.750704 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.755919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.755863 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.762669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.762641 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dv2\" (UniqueName: \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\") pod \"maas-api-revoke-sub-3109ca2f-8tczt\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:34.896955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.896859 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" event={"ID":"4228e118-12fe-4495-83de-10dd93af169b","Type":"ContainerStarted","Data":"24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0"} Oct 30 22:17:34.955600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:34.954497 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:35.137314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.134333 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:35.262049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.261955 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\") pod \"d89dd5d3-446d-47e4-a65a-8462a4730638\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " Oct 30 22:17:35.262200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.262048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\" (UniqueName: \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\") pod \"d89dd5d3-446d-47e4-a65a-8462a4730638\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " Oct 30 22:17:35.262200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.262110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\") pod \"d89dd5d3-446d-47e4-a65a-8462a4730638\" (UID: \"d89dd5d3-446d-47e4-a65a-8462a4730638\") " Oct 30 22:17:35.262389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.262339 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca" pod "d89dd5d3-446d-47e4-a65a-8462a4730638" (UID: "d89dd5d3-446d-47e4-a65a-8462a4730638"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:35.264367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.263009 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:35.264957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.264934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5" pod "d89dd5d3-446d-47e4-a65a-8462a4730638" (UID: "d89dd5d3-446d-47e4-a65a-8462a4730638"). InnerVolumeSpecName "kube-api-access-5nrc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:35.265209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.265193 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca" pod "d89dd5d3-446d-47e4-a65a-8462a4730638" (UID: "d89dd5d3-446d-47e4-a65a-8462a4730638"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:35.348804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.348772 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt"] Oct 30 22:17:35.349035 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:35.349008 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d826f90_f0f0_497f_92d7_93524eb290c5.slice/crio-64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0 WatchSource:0}: Error finding container 64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0: Status 404 returned error can't find the container with id 64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0 Oct 30 22:17:35.365033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.365005 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89dd5d3-446d-47e4-a65a-8462a4730638-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:35.365152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.365040 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrc5\" (UniqueName: \"kubernetes.io/projected/d89dd5d3-446d-47e4-a65a-8462a4730638-kube-api-access-5nrc5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:35.907127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.907092 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" Oct 30 22:17:35.907537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.907129 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c" event={"ID":"d89dd5d3-446d-47e4-a65a-8462a4730638","Type":"ContainerDied","Data":"ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416"} Oct 30 22:17:35.907537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.907186 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced85c4aa9aec1a51ba0b6404c2c2ae56db8df29b88c33ee03eb25e744821416" Oct 30 22:17:35.909086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.909054 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" event={"ID":"4228e118-12fe-4495-83de-10dd93af169b","Type":"ContainerStarted","Data":"226557496bcdeefa580e95450121a3e3003f6ac2ce041331f06b150cf67a265e"} Oct 30 22:17:35.911278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.911241 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" event={"ID":"7d826f90-f0f0-497f-92d7-93524eb290c5","Type":"ContainerStarted","Data":"6c7c86d6fbb55ae402f9c04384332cbcf17d6f08fb28880e4c371426eb24241c"} Oct 30 22:17:35.911392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.911304 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" event={"ID":"7d826f90-f0f0-497f-92d7-93524eb290c5","Type":"ContainerStarted","Data":"64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0"} Oct 30 22:17:35.932509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.932463 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" podStartSLOduration=2.932442156 podStartE2EDuration="2.932442156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:35.929830721 +0000 UTC m=+2279.879785667" watchObservedRunningTime="2026-10-30 22:17:35.932442156 +0000 UTC m=+2279.882397101" Oct 30 22:17:35.955959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:35.955911 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" podStartSLOduration=1.95589491 podStartE2EDuration="1.95589491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:35.95525497 +0000 UTC m=+2279.905209916" watchObservedRunningTime="2026-10-30 22:17:35.95589491 +0000 UTC m=+2279.905849857" Oct 30 22:17:36.925044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:36.925006 5036 generic.go:356] "Generic (PLEG): container finished" podID="7d826f90-f0f0-497f-92d7-93524eb290c5" containerID="6c7c86d6fbb55ae402f9c04384332cbcf17d6f08fb28880e4c371426eb24241c" exitCode=0 Oct 30 22:17:36.925576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:36.925091 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" event={"ID":"7d826f90-f0f0-497f-92d7-93524eb290c5","Type":"ContainerDied","Data":"6c7c86d6fbb55ae402f9c04384332cbcf17d6f08fb28880e4c371426eb24241c"} Oct 30 22:17:36.927428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:36.927401 5036 generic.go:356] "Generic (PLEG): container finished" podID="4228e118-12fe-4495-83de-10dd93af169b" containerID="226557496bcdeefa580e95450121a3e3003f6ac2ce041331f06b150cf67a265e" exitCode=0 Oct 30 22:17:36.927559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:36.927486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" event={"ID":"4228e118-12fe-4495-83de-10dd93af169b","Type":"ContainerDied","Data":"226557496bcdeefa580e95450121a3e3003f6ac2ce041331f06b150cf67a265e"} Oct 30 22:17:38.189816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.189746 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:38.264781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.264747 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:38.352889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.352849 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\" (UniqueName: \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\") pod \"4228e118-12fe-4495-83de-10dd93af169b\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " Oct 30 22:17:38.353066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.352916 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\") pod \"4228e118-12fe-4495-83de-10dd93af169b\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " Oct 30 22:17:38.353304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.353279 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\") pod \"4228e118-12fe-4495-83de-10dd93af169b\" (UID: \"4228e118-12fe-4495-83de-10dd93af169b\") " Oct 30 22:17:38.355488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.355454 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca" pod "4228e118-12fe-4495-83de-10dd93af169b" (UID: "4228e118-12fe-4495-83de-10dd93af169b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:38.358451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.358415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca" pod "4228e118-12fe-4495-83de-10dd93af169b" (UID: "4228e118-12fe-4495-83de-10dd93af169b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:38.358581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.358466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb" pod "4228e118-12fe-4495-83de-10dd93af169b" (UID: "4228e118-12fe-4495-83de-10dd93af169b"). InnerVolumeSpecName "kube-api-access-wfcxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:38.454645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.454561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\" (UniqueName: \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\") pod \"7d826f90-f0f0-497f-92d7-93524eb290c5\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " Oct 30 22:17:38.454823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.454772 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\") pod \"7d826f90-f0f0-497f-92d7-93524eb290c5\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " Oct 30 22:17:38.454892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.454832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\") pod \"7d826f90-f0f0-497f-92d7-93524eb290c5\" (UID: \"7d826f90-f0f0-497f-92d7-93524eb290c5\") " Oct 30 22:17:38.455310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.455160 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca" pod "7d826f90-f0f0-497f-92d7-93524eb290c5" (UID: "7d826f90-f0f0-497f-92d7-93524eb290c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:38.455595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.455572 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4228e118-12fe-4495-83de-10dd93af169b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.455707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.455603 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcxb\" (UniqueName: \"kubernetes.io/projected/4228e118-12fe-4495-83de-10dd93af169b-kube-api-access-wfcxb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.455707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.455621 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4228e118-12fe-4495-83de-10dd93af169b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.455707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.455636 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.457134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.457107 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca" pod "7d826f90-f0f0-497f-92d7-93524eb290c5" (UID: "7d826f90-f0f0-497f-92d7-93524eb290c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:38.457454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.457421 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2" pod "7d826f90-f0f0-497f-92d7-93524eb290c5" (UID: "7d826f90-f0f0-497f-92d7-93524eb290c5"). InnerVolumeSpecName "kube-api-access-c5dv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:38.557284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.557233 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d826f90-f0f0-497f-92d7-93524eb290c5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.557284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.557284 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5dv2\" (UniqueName: \"kubernetes.io/projected/7d826f90-f0f0-497f-92d7-93524eb290c5-kube-api-access-c5dv2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.949758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.949721 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" Oct 30 22:17:38.949758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.949745 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd" event={"ID":"4228e118-12fe-4495-83de-10dd93af169b","Type":"ContainerDied","Data":"24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0"} Oct 30 22:17:38.949979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.949806 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a5d49541909e4f2feebe3c3cad9c8039cade0ff11baab65d18b0477bbbe1a0" Oct 30 22:17:38.951555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.951535 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" Oct 30 22:17:38.951712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.951573 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt" event={"ID":"7d826f90-f0f0-497f-92d7-93524eb290c5","Type":"ContainerDied","Data":"64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0"} Oct 30 22:17:38.951712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:38.951620 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ff818a7fa6ae68693d479eb71aa00c355e921cb21654befad33700bad073f0" Oct 30 22:17:43.257573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:43.257533 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:17:45.671245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.671213 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:17:45.672106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672087 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7d826f90-f0f0-497f-92d7-93524eb290c5" containerName="revoke-keys" Oct 30 22:17:45.672168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672108 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d826f90-f0f0-497f-92d7-93524eb290c5" containerName="revoke-keys" Oct 30 22:17:45.672168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672122 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d89dd5d3-446d-47e4-a65a-8462a4730638" containerName="revoke-keys" Oct 30 22:17:45.672168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672127 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89dd5d3-446d-47e4-a65a-8462a4730638" containerName="revoke-keys" Oct 30 22:17:45.672168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672136 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4228e118-12fe-4495-83de-10dd93af169b" containerName="revoke-keys" Oct 30 22:17:45.672168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672141 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4228e118-12fe-4495-83de-10dd93af169b" containerName="revoke-keys" Oct 30 22:17:45.672359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672284 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d826f90-f0f0-497f-92d7-93524eb290c5" containerName="revoke-keys" Oct 30 22:17:45.672359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672301 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89dd5d3-446d-47e4-a65a-8462a4730638" containerName="revoke-keys" Oct 30 22:17:45.672359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.672311 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4228e118-12fe-4495-83de-10dd93af169b" containerName="revoke-keys" Oct 30 22:17:45.676818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.676797 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.684585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.684555 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-af383a1b829d\"" Oct 30 22:17:45.684718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.684636 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hpdgn\"" Oct 30 22:17:45.686304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.686104 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 22:17:45.686304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.686181 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 22:17:45.686304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.686223 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 22:17:45.686304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.686240 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 22:17:45.689336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.689312 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:17:45.823456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.823360 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.823643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.823496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.823643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.823545 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.823643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.823572 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkkq\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.926591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.926493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.927554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.927519 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.932372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.932348 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.932540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.932522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkkq\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.937285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.928421 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.937471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.934286 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.960938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.954884 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkkq\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:45.967864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:45.967840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\") pod \"payload-processing-external-model-7cbb97f857-xv7kk\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:46.004866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.004592 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:46.656144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.656103 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:17:46.663447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.663420 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.665923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.665897 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:17:46.666927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.666898 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"agc-cred-3ae043299db1\"" Oct 30 22:17:46.667354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.667327 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"openshift-service-ca.crt\"" Oct 30 22:17:46.667453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.667394 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"kube-root-ca.crt\"" Oct 30 22:17:46.667519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.667497 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1a5bdf\"" Oct 30 22:17:46.667595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.667579 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"payload-processing-external-model-e2e--378e44ca-dockercfg-d5nzw\"" Oct 30 22:17:46.668097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.668079 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"routing-overlay\"" Oct 30 22:17:46.709961 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:46.709932 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8e6cfe_6612_415e_ae4d_b6d3bc29dfc4.slice/crio-7aa37f71f3f69b7c4eebef4b48177225c438354d6f9ecdf5db3c29a990be0caa WatchSource:0}: Error finding container 7aa37f71f3f69b7c4eebef4b48177225c438354d6f9ecdf5db3c29a990be0caa: Status 404 returned error can't find the container with id 7aa37f71f3f69b7c4eebef4b48177225c438354d6f9ecdf5db3c29a990be0caa Oct 30 22:17:46.710633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.710611 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:17:46.800298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.791604 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.800298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.791772 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjhc\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.800298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.791887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.800298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.791957 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.897292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.893809 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.897292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.893968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjhc\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.897292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.894089 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.897292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.894154 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.897292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.895884 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.914289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.911999 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:46.918292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.915725 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99"] Oct 30 22:17:46.936299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:46.920351 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:47.601379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.601196 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99"] Oct 30 22:17:47.619062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.606844 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjhc\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:47.620893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.620865 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:47.650098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.650072 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"routing-overlay\"" Oct 30 22:17:47.668423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.668393 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-17e7bc508a71\"" Oct 30 22:17:47.668720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.668702 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"openshift-service-ca.crt\"" Oct 30 22:17:47.668949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.668933 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"kube-root-ca.crt\"" Oct 30 22:17:47.669127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.669113 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f8222\"" Oct 30 22:17:47.672320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.668037 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-cb4046ae8a86\"" Oct 30 22:17:47.719291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.668104 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-e2e--96f1cc57-dockercfg-5bqlg\"" Oct 30 22:17:47.827877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.824587 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:47.832623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.832597 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:47.870057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.870026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" event={"ID":"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4","Type":"ContainerStarted","Data":"7aa37f71f3f69b7c4eebef4b48177225c438354d6f9ecdf5db3c29a990be0caa"} Oct 30 22:17:47.981886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.979953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5978\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:47.981886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.980123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:47.983234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.980255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:47.983234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:47.982361 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.020254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.020218 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6"] Oct 30 22:17:48.028964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.028939 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.048426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.033342 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:17:48.048426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.033401 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:48.048426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.036010 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6"] Oct 30 22:17:48.069345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.068705 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5978\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091353 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091423 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091488 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091584 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.091660 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdmx\" (UniqueName: \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.092346 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6f8222/agc-cred-cb4046ae8a86: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.092367 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.092425 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials podName:7160a099-4c39-466d-b856-3fa6fb1ecfa1 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:48.59240382 +0000 UTC m=+2292.542358767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1") : secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.092612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.092549 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.093386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.092979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.109207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.109132 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5978\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.140222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.140179 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.261337 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerName="payload-processing" containerID="cri-o://ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24" gracePeriod=25 Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.266233 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.266838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdmx\" (UniqueName: \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.267119 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.267408 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.292421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.278979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.294394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.293454 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.303292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.300981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdmx\" (UniqueName: \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\") pod \"maas-api-revoke-sub-8cdb0ea5-tpkd6\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.435216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.425843 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp"] Oct 30 22:17:48.435216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.432991 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:48.439105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.439078 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.449920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.449881 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp"] Oct 30 22:17:48.493705 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:48.493683 5036 logging.go:55] [core] [Channel #9724 SubChannel #9725]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.75:50052", ServerName: "10.132.1.75:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.75:50052: connect: connection refused" Oct 30 22:17:48.509167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.509141 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.641486 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.641542 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lls2l\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.641587 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.641708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.641759 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.642029 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6f8222/agc-cred-cb4046ae8a86: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.642047 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.642614 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.642107 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials podName:7160a099-4c39-466d-b856-3fa6fb1ecfa1 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:49.642086941 +0000 UTC m=+2293.592041878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1") : secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:48.688415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.688390 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:17:48.743535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.743486 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\") pod \"18a23f99-39db-4334-8644-b5f0b9ad237c\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " Oct 30 22:17:48.744039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.743705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\") pod \"18a23f99-39db-4334-8644-b5f0b9ad237c\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " Oct 30 22:17:48.744039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.743808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\") pod \"18a23f99-39db-4334-8644-b5f0b9ad237c\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " Oct 30 22:17:48.744039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.743848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\") pod \"18a23f99-39db-4334-8644-b5f0b9ad237c\" (UID: \"18a23f99-39db-4334-8644-b5f0b9ad237c\") " Oct 30 22:17:48.744300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744235 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume" pod "18a23f99-39db-4334-8644-b5f0b9ad237c" (UID: "18a23f99-39db-4334-8644-b5f0b9ad237c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:48.744395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744304 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.744395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lls2l\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.744395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744378 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.744589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744526 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.744676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744659 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:48.744738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.744687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume" pod "18a23f99-39db-4334-8644-b5f0b9ad237c" (UID: "18a23f99-39db-4334-8644-b5f0b9ad237c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:48.745583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.745557 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.745727 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.745660 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-277a231d8375: secret "agc-cred-277a231d8375" not found Oct 30 22:17:48.745727 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.745680 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp: secret "agc-cred-277a231d8375" not found Oct 30 22:17:48.745866 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.745745 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials podName:c8ee6275-99dd-4fc9-9566-0531f3c54947 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:49.245727116 +0000 UTC m=+2293.195682055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947") : secret "agc-cred-277a231d8375" not found Oct 30 22:17:48.745866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.745749 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.746784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.746756 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg" pod "18a23f99-39db-4334-8644-b5f0b9ad237c" (UID: "18a23f99-39db-4334-8644-b5f0b9ad237c"). InnerVolumeSpecName "kube-api-access-fq9dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:48.748507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.748480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials" pod "18a23f99-39db-4334-8644-b5f0b9ad237c" (UID: "18a23f99-39db-4334-8644-b5f0b9ad237c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:48.757025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.756995 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lls2l\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:48.768767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.768740 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6"] Oct 30 22:17:48.769082 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:48.769041 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a38797_5d90_411a_b8bd_1eace3fd6eb2.slice/crio-4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50 WatchSource:0}: Error finding container 4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50: Status 404 returned error can't find the container with id 4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50 Oct 30 22:17:48.846064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.846029 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:48.846064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.846063 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9dg\" (UniqueName: \"kubernetes.io/projected/18a23f99-39db-4334-8644-b5f0b9ad237c-kube-api-access-fq9dg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:48.846256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.846079 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a23f99-39db-4334-8644-b5f0b9ad237c-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:48.880216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.880176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" event={"ID":"49a38797-5d90-411a-b8bd-1eace3fd6eb2","Type":"ContainerStarted","Data":"4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50"} Oct 30 22:17:48.881982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.881951 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" event={"ID":"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4","Type":"ContainerStarted","Data":"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75"} Oct 30 22:17:48.882120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.882006 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:48.883858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.883833 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" event={"ID":"b7aac589-0369-4952-be8b-5ee693d6db8b","Type":"ContainerStarted","Data":"48358710faddc71083977ce33b8753fb8127ed9997df8e078112cbaa53b98cc5"} Oct 30 22:17:48.883958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.883876 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" event={"ID":"b7aac589-0369-4952-be8b-5ee693d6db8b","Type":"ContainerStarted","Data":"2b230578f822ac5b3c2c56d0ad06c758e912985fd2ad5c24152604a1d4230a8c"} Oct 30 22:17:48.883958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.883932 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:48.885367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.885342 5036 generic.go:356] "Generic (PLEG): container finished" podID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerID="ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24" exitCode=0 Oct 30 22:17:48.885461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.885379 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" Oct 30 22:17:48.885461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.885402 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" event={"ID":"18a23f99-39db-4334-8644-b5f0b9ad237c","Type":"ContainerDied","Data":"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24"} Oct 30 22:17:48.885461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.885441 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" event={"ID":"18a23f99-39db-4334-8644-b5f0b9ad237c","Type":"ContainerDied","Data":"125b3ec64d01dcfb691025dac7a9a7237bf910041f62e633b13e7dc0d50925f6"} Oct 30 22:17:48.885589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.885462 5036 scope.go:122] "RemoveContainer" containerID="ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24" Oct 30 22:17:48.897146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.897131 5036 scope.go:122] "RemoveContainer" containerID="ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24" Oct 30 22:17:48.897389 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:48.897366 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24\": container with ID starting with ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24 not found: ID does not exist" containerID="ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24" Oct 30 22:17:48.897445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.897402 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24"} err="failed to get container status \"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24\": rpc error: code = NotFound desc = could not find container \"ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24\": container with ID starting with ff77f6624b134a55244b59e5de936f54eb3cbf532ca2f17318b345f1872dde24 not found: ID does not exist" Oct 30 22:17:48.903053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.903008 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" podStartSLOduration=3.902992757 podStartE2EDuration="3.902992757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:48.899154934 +0000 UTC m=+2292.849109880" watchObservedRunningTime="2026-10-30 22:17:48.902992757 +0000 UTC m=+2292.852947706" Oct 30 22:17:48.918809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.918784 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl"] Oct 30 22:17:48.920839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.920814 5036 cpu_manager.go:414] "containerMap: removing container" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerName="payload-processing" Oct 30 22:17:48.920971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.920840 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerName="payload-processing" Oct 30 22:17:48.921164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.921148 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerName="payload-processing" Oct 30 22:17:48.930233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.930179 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:17:48.930525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.930327 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:48.934973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.933934 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj"] Oct 30 22:17:48.945416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.942106 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl"] Oct 30 22:17:48.945416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.943102 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" podStartSLOduration=2.943086444 podStartE2EDuration="2.943086444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:48.939904495 +0000 UTC m=+2292.889859441" watchObservedRunningTime="2026-10-30 22:17:48.943086444 +0000 UTC m=+2292.893041407" Oct 30 22:17:48.947910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.947882 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdqv\" (UniqueName: \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:48.948162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.948139 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:48.948477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:48.948456 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.055290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.050701 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.055290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.050848 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.055290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.051218 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdqv\" (UniqueName: \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.055290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.052134 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.088285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.086774 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.115644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.114799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdqv\" (UniqueName: \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\") pod \"maas-api-revoke-sub-5ab8bd72-srfkl\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.255237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.255139 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:49.255471 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.255307 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-277a231d8375: secret "agc-cred-277a231d8375" not found Oct 30 22:17:49.255471 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.255333 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp: secret "agc-cred-277a231d8375" not found Oct 30 22:17:49.255471 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.255403 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials podName:c8ee6275-99dd-4fc9-9566-0531f3c54947 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:50.255382681 +0000 UTC m=+2294.205337605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947") : secret "agc-cred-277a231d8375" not found Oct 30 22:17:49.255758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.255729 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:49.257602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.257573 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99"] Oct 30 22:17:49.258074 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.258047 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" podUID="7160a099-4c39-466d-b856-3fa6fb1ecfa1" Oct 30 22:17:49.288309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.288281 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:17:49.294504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.294482 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.305136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.305113 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:17:49.358045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.358024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.358386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.358370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.358630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.358618 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.358759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.358749 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwppd\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.475548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.475512 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.475814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.475789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.476031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.475860 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwppd\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.476031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.475955 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.477344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.477014 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.477344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.477034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.481679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.481627 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.490899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.490835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwppd\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.492703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.492674 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-6649xghrj" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.75:50052\" within 1s: context deadline exceeded" Oct 30 22:17:49.573654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.573625 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl"] Oct 30 22:17:49.574993 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:49.574967 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfe6766_bd59_4b96_89fe_b0829a695155.slice/crio-6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639 WatchSource:0}: Error finding container 6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639: Status 404 returned error can't find the container with id 6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639 Oct 30 22:17:49.621830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.621803 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:49.677739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.677694 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp"] Oct 30 22:17:49.678505 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.678467 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" podUID="c8ee6275-99dd-4fc9-9566-0531f3c54947" Oct 30 22:17:49.678794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.678773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:49.678955 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.678937 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6f8222/agc-cred-cb4046ae8a86: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:49.678993 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.678961 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99: secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:49.679059 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.679028 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials podName:7160a099-4c39-466d-b856-3fa6fb1ecfa1 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:51.679007504 +0000 UTC m=+2295.628962447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1") : secret "agc-cred-cb4046ae8a86" not found Oct 30 22:17:49.723154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.723119 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:17:49.740557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.740530 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:17:49.740892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.740825 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.779856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.779827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.780321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.780054 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.780321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.780210 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.780476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.780438 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.865013 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:49.864973 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a38797_5d90_411a_b8bd_1eace3fd6eb2.slice/crio-3ca7c8d4c9bbfacf0122c1db6e858fb4d32bfb42e4365638f086060373764aff.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:49.882225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.882198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.882440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.882325 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.882440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.882435 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.882574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.882534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.883066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.883039 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.883161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.883073 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.885012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.884993 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.894278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.894235 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:49.896541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.896513 5036 generic.go:356] "Generic (PLEG): container finished" podID="49a38797-5d90-411a-b8bd-1eace3fd6eb2" containerID="3ca7c8d4c9bbfacf0122c1db6e858fb4d32bfb42e4365638f086060373764aff" exitCode=0 Oct 30 22:17:49.896652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.896599 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" event={"ID":"49a38797-5d90-411a-b8bd-1eace3fd6eb2","Type":"ContainerDied","Data":"3ca7c8d4c9bbfacf0122c1db6e858fb4d32bfb42e4365638f086060373764aff"} Oct 30 22:17:49.898160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.898135 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" event={"ID":"ebfe6766-bd59-4b96-89fe-b0829a695155","Type":"ContainerStarted","Data":"6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639"} Oct 30 22:17:49.899407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.899384 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:49.899735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.899717 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:49.908316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.908255 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:49.912260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.912240 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:49.928454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.928424 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:17:49.928799 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:49.928778 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d448d61_91ae_47ab_a1c4_adcd968b3da2.slice/crio-c3fa2b18285c1adecafb05199471ff84b204514f2fb1cc158934cf7ceafe6fd2 WatchSource:0}: Error finding container c3fa2b18285c1adecafb05199471ff84b204514f2fb1cc158934cf7ceafe6fd2: Status 404 returned error can't find the container with id c3fa2b18285c1adecafb05199471ff84b204514f2fb1cc158934cf7ceafe6fd2 Oct 30 22:17:49.983702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.983664 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\") pod \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " Oct 30 22:17:49.983820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.983783 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\") pod \"c8ee6275-99dd-4fc9-9566-0531f3c54947\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " Oct 30 22:17:49.984080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984016 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\") pod \"c8ee6275-99dd-4fc9-9566-0531f3c54947\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " Oct 30 22:17:49.984080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984051 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\") pod \"c8ee6275-99dd-4fc9-9566-0531f3c54947\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " Oct 30 22:17:49.984247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984151 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume" pod "7160a099-4c39-466d-b856-3fa6fb1ecfa1" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:49.984247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984152 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume" pod "c8ee6275-99dd-4fc9-9566-0531f3c54947" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:49.984567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984543 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume" pod "c8ee6275-99dd-4fc9-9566-0531f3c54947" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:49.984651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984588 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume" pod "7160a099-4c39-466d-b856-3fa6fb1ecfa1" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:49.984651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984603 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\") pod \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " Oct 30 22:17:49.984760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.984679 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\") pod \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\" (UID: \"7160a099-4c39-466d-b856-3fa6fb1ecfa1\") " Oct 30 22:17:49.986044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.985961 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:49.986044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.985990 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:49.986044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.986006 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7160a099-4c39-466d-b856-3fa6fb1ecfa1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:49.986044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.986020 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee6275-99dd-4fc9-9566-0531f3c54947-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:49.987117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.987096 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l" pod "c8ee6275-99dd-4fc9-9566-0531f3c54947" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947"). InnerVolumeSpecName "kube-api-access-lls2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:49.987425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:49.987406 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978" pod "7160a099-4c39-466d-b856-3fa6fb1ecfa1" (UID: "7160a099-4c39-466d-b856-3fa6fb1ecfa1"). InnerVolumeSpecName "kube-api-access-l5978". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:50.066138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.066106 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:50.087752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.087723 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lls2l\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-kube-api-access-lls2l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:50.087840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.087753 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5978\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-kube-api-access-l5978\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:50.205542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.205505 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a23f99-39db-4334-8644-b5f0b9ad237c" path="/var/lib/kubelet/pods/18a23f99-39db-4334-8644-b5f0b9ad237c/volumes" Oct 30 22:17:50.290610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.290570 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp\" (UID: \"c8ee6275-99dd-4fc9-9566-0531f3c54947\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:50.290775 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:50.290727 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-277a231d8375: secret "agc-cred-277a231d8375" not found Oct 30 22:17:50.290775 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:50.290751 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp: secret "agc-cred-277a231d8375" not found Oct 30 22:17:50.290865 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:17:50.290809 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials podName:c8ee6275-99dd-4fc9-9566-0531f3c54947 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:52.290794451 +0000 UTC m=+2296.240749385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" (UID: "c8ee6275-99dd-4fc9-9566-0531f3c54947") : secret "agc-cred-277a231d8375" not found Oct 30 22:17:50.419634 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:17:50.419588 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a57585_851e_4269_83b1_23b7b478634d.slice/crio-06fb208ee5c921dffec2a4a420c1dc5035ae722432e4a6b519416ddead1d7f0d WatchSource:0}: Error finding container 06fb208ee5c921dffec2a4a420c1dc5035ae722432e4a6b519416ddead1d7f0d: Status 404 returned error can't find the container with id 06fb208ee5c921dffec2a4a420c1dc5035ae722432e4a6b519416ddead1d7f0d Oct 30 22:17:50.421823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.421791 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:17:50.914870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.914835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" event={"ID":"9d448d61-91ae-47ab-a1c4-adcd968b3da2","Type":"ContainerStarted","Data":"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77"} Oct 30 22:17:50.915479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.914895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" event={"ID":"9d448d61-91ae-47ab-a1c4-adcd968b3da2","Type":"ContainerStarted","Data":"c3fa2b18285c1adecafb05199471ff84b204514f2fb1cc158934cf7ceafe6fd2"} Oct 30 22:17:50.915479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.914972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:17:50.917152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.917121 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" event={"ID":"f1a57585-851e-4269-83b1-23b7b478634d","Type":"ContainerStarted","Data":"c782e1cd76ddbc9d2f0511793498017e12e745e9f8742083523eee5f1115b7a1"} Oct 30 22:17:50.917304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.917162 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" event={"ID":"f1a57585-851e-4269-83b1-23b7b478634d","Type":"ContainerStarted","Data":"06fb208ee5c921dffec2a4a420c1dc5035ae722432e4a6b519416ddead1d7f0d"} Oct 30 22:17:50.917304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.917199 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:17:50.919188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.919158 5036 generic.go:356] "Generic (PLEG): container finished" podID="ebfe6766-bd59-4b96-89fe-b0829a695155" containerID="ac4d6ed6dab5130eed846fb89f86eee1da4c93a7ebd131eeefecdfdcbe4569bf" exitCode=0 Oct 30 22:17:50.919366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.919237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" event={"ID":"ebfe6766-bd59-4b96-89fe-b0829a695155","Type":"ContainerDied","Data":"ac4d6ed6dab5130eed846fb89f86eee1da4c93a7ebd131eeefecdfdcbe4569bf"} Oct 30 22:17:50.919596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.919537 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99" Oct 30 22:17:50.919596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.919541 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp" Oct 30 22:17:50.938136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.937428 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" podStartSLOduration=1.93741071 podStartE2EDuration="1.93741071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:50.936182785 +0000 UTC m=+2294.886137731" watchObservedRunningTime="2026-10-30 22:17:50.93741071 +0000 UTC m=+2294.887365657" Oct 30 22:17:50.990981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.990954 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp"] Oct 30 22:17:50.996960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:50.996938 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5c59cvjkp"] Oct 30 22:17:51.029345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.029315 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99"] Oct 30 22:17:51.037744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.037715 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-6f4f86g99"] Oct 30 22:17:51.048016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.046746 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" podStartSLOduration=2.046730114 podStartE2EDuration="2.046730114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:51.045833153 +0000 UTC m=+2294.995788099" watchObservedRunningTime="2026-10-30 22:17:51.046730114 +0000 UTC m=+2294.996685060" Oct 30 22:17:51.105147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.105117 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7160a099-4c39-466d-b856-3fa6fb1ecfa1-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.105147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.105146 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8ee6275-99dd-4fc9-9566-0531f3c54947-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.142694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.142661 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:51.205942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.205901 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\") pod \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " Oct 30 22:17:51.206138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.205952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\" (UniqueName: \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\") pod \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " Oct 30 22:17:51.206311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.206282 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca" pod "49a38797-5d90-411a-b8bd-1eace3fd6eb2" (UID: "49a38797-5d90-411a-b8bd-1eace3fd6eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:51.206311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.206256 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\") pod \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\" (UID: \"49a38797-5d90-411a-b8bd-1eace3fd6eb2\") " Oct 30 22:17:51.207095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.207070 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.208532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.208506 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx" pod "49a38797-5d90-411a-b8bd-1eace3fd6eb2" (UID: "49a38797-5d90-411a-b8bd-1eace3fd6eb2"). InnerVolumeSpecName "kube-api-access-wkdmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:51.208664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.208561 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca" pod "49a38797-5d90-411a-b8bd-1eace3fd6eb2" (UID: "49a38797-5d90-411a-b8bd-1eace3fd6eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:51.308371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.308288 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdmx\" (UniqueName: \"kubernetes.io/projected/49a38797-5d90-411a-b8bd-1eace3fd6eb2-kube-api-access-wkdmx\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.308371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.308327 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a38797-5d90-411a-b8bd-1eace3fd6eb2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.936342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.936299 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" event={"ID":"49a38797-5d90-411a-b8bd-1eace3fd6eb2","Type":"ContainerDied","Data":"4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50"} Oct 30 22:17:51.936786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.936367 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e0adb3242efb68ae2fc2c1c5a0029fb75e0fd3ac8cd394ed6099787fb759b50" Oct 30 22:17:51.936786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:51.936367 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6" Oct 30 22:17:52.149855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.149821 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:52.196759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.196680 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7160a099-4c39-466d-b856-3fa6fb1ecfa1" path="/var/lib/kubelet/pods/7160a099-4c39-466d-b856-3fa6fb1ecfa1/volumes" Oct 30 22:17:52.197055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.197041 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ee6275-99dd-4fc9-9566-0531f3c54947" path="/var/lib/kubelet/pods/c8ee6275-99dd-4fc9-9566-0531f3c54947/volumes" Oct 30 22:17:52.268333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.228161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\" (UniqueName: \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\") pod \"ebfe6766-bd59-4b96-89fe-b0829a695155\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " Oct 30 22:17:52.268333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.228358 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\") pod \"ebfe6766-bd59-4b96-89fe-b0829a695155\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " Oct 30 22:17:52.268333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.228420 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\") pod \"ebfe6766-bd59-4b96-89fe-b0829a695155\" (UID: \"ebfe6766-bd59-4b96-89fe-b0829a695155\") " Oct 30 22:17:52.268333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.264501 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca" pod "ebfe6766-bd59-4b96-89fe-b0829a695155" (UID: "ebfe6766-bd59-4b96-89fe-b0829a695155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:52.291549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.291484 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv" pod "ebfe6766-bd59-4b96-89fe-b0829a695155" (UID: "ebfe6766-bd59-4b96-89fe-b0829a695155"). InnerVolumeSpecName "kube-api-access-kjdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:52.296324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.292549 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca" pod "ebfe6766-bd59-4b96-89fe-b0829a695155" (UID: "ebfe6766-bd59-4b96-89fe-b0829a695155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:52.331076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.331042 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdqv\" (UniqueName: \"kubernetes.io/projected/ebfe6766-bd59-4b96-89fe-b0829a695155-kube-api-access-kjdqv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.331076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.331073 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.331313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.331084 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfe6766-bd59-4b96-89fe-b0829a695155-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.949109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.949076 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" Oct 30 22:17:52.949109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.949099 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl" event={"ID":"ebfe6766-bd59-4b96-89fe-b0829a695155","Type":"ContainerDied","Data":"6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639"} Oct 30 22:17:52.949578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:52.949159 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6694fe877a3a0ce67925f6d2edca538ef57ad0f5ef58b86bddc0b695a87bb639" Oct 30 22:17:53.650680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.650645 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:17:53.651925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.651906 5036 cpu_manager.go:414] "containerMap: removing container" podUID="49a38797-5d90-411a-b8bd-1eace3fd6eb2" containerName="revoke-keys" Oct 30 22:17:53.651973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.651929 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a38797-5d90-411a-b8bd-1eace3fd6eb2" containerName="revoke-keys" Oct 30 22:17:53.652038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.652027 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ebfe6766-bd59-4b96-89fe-b0829a695155" containerName="revoke-keys" Oct 30 22:17:53.652080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.652038 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfe6766-bd59-4b96-89fe-b0829a695155" containerName="revoke-keys" Oct 30 22:17:53.652249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.652236 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a38797-5d90-411a-b8bd-1eace3fd6eb2" containerName="revoke-keys" Oct 30 22:17:53.652295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.652289 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfe6766-bd59-4b96-89fe-b0829a695155" containerName="revoke-keys" Oct 30 22:17:53.658795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.658773 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.665382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.665354 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:17:53.667443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.667354 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-17b4e6be43d5\"" Oct 30 22:17:53.764064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.764020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.764241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.764094 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.764346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.764242 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.764346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.764309 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnlp\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.866215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.866173 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.866460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.866247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.866460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.866417 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.866583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.866461 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnlp\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.867132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.867066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.867132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.867075 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.869014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.868992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.878362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.878341 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnlp\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\") pod \"payload-processing-external-model-84ff676cfc-svfjk\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:53.978446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:53.978360 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:54.302909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:54.302876 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:17:54.987649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:54.987607 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" event={"ID":"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3","Type":"ContainerStarted","Data":"227b8da3b3769efcdff6518de24a60cd3f7c99906a3c6714315a176d68a10b59"} Oct 30 22:17:54.988128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:54.987679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" event={"ID":"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3","Type":"ContainerStarted","Data":"2980a8871e0a76a2a0e78e5e00f3c60735492c70703640c16186ee73cb7a5c78"} Oct 30 22:17:55.008257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:55.008203 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" podStartSLOduration=2.008185584 podStartE2EDuration="2.008185584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:55.005998534 +0000 UTC m=+2298.955953480" watchObservedRunningTime="2026-10-30 22:17:55.008185584 +0000 UTC m=+2298.958140530" Oct 30 22:17:55.997291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:55.997243 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:17:59.903167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:59.903134 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:17:59.903694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:59.903618 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:17:59.981861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:59.979483 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:17:59.989076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:17:59.989043 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:18:00.026459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.026428 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:18:00.609863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.609831 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:00.615112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.615089 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.621942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.621906 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-56549985aa0b\"" Oct 30 22:18:00.621942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.621915 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-e2e--2051f056-dockercfg-8pjcm\"" Oct 30 22:18:00.626843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.626821 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:00.634216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.633998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.634216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.634081 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.634436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.634223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.634436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.634307 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjphc\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.745038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.742698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.745038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.742778 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.745038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.742913 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.745038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.742975 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjphc\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.771284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.771223 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.837324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.837286 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.893317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.862151 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:18:00.981296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.970813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:00.990311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:00.989927 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:18:01.051126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:01.051085 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjphc\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:01.255961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:01.255875 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:01.757690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:01.750205 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:02.242218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:02.242182 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:02.301357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:02.301316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" event={"ID":"d0eea665-1822-43ca-9034-2da89c63a415","Type":"ContainerStarted","Data":"7e7e0adb2e238ab43b02c5f7b294e0c79969b6870455b229d924277a162a479d"} Oct 30 22:18:03.313199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:03.313160 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" event={"ID":"d0eea665-1822-43ca-9034-2da89c63a415","Type":"ContainerStarted","Data":"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1"} Oct 30 22:18:03.313715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:03.313227 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:03.337201 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:03.337136 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" podStartSLOduration=3.337117987 podStartE2EDuration="3.337117987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:03.330981098 +0000 UTC m=+2307.280936051" watchObservedRunningTime="2026-10-30 22:18:03.337117987 +0000 UTC m=+2307.287072939" Oct 30 22:18:04.980770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:04.980722 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" podUID="695563f5-e441-4538-b5b2-5617da6965b0" containerName="payload-processing" containerID="cri-o://f2baa6bf5f7fa5c40a08cf54caf8de09724e806523e1f95de0a5579d01185716" gracePeriod=25 Oct 30 22:18:04.990166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:04.990117 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" podUID="f1a57585-851e-4269-83b1-23b7b478634d" containerName="payload-processing" containerID="cri-o://c782e1cd76ddbc9d2f0511793498017e12e745e9f8742083523eee5f1115b7a1" gracePeriod=25 Oct 30 22:18:05.340099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.339897 5036 generic.go:356] "Generic (PLEG): container finished" podID="695563f5-e441-4538-b5b2-5617da6965b0" containerID="f2baa6bf5f7fa5c40a08cf54caf8de09724e806523e1f95de0a5579d01185716" exitCode=0 Oct 30 22:18:05.340099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.340037 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" event={"ID":"695563f5-e441-4538-b5b2-5617da6965b0","Type":"ContainerDied","Data":"f2baa6bf5f7fa5c40a08cf54caf8de09724e806523e1f95de0a5579d01185716"} Oct 30 22:18:05.342870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.342841 5036 generic.go:356] "Generic (PLEG): container finished" podID="f1a57585-851e-4269-83b1-23b7b478634d" containerID="c782e1cd76ddbc9d2f0511793498017e12e745e9f8742083523eee5f1115b7a1" exitCode=0 Oct 30 22:18:05.343450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.343414 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" event={"ID":"f1a57585-851e-4269-83b1-23b7b478634d","Type":"ContainerDied","Data":"c782e1cd76ddbc9d2f0511793498017e12e745e9f8742083523eee5f1115b7a1"} Oct 30 22:18:05.436513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.436483 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:18:05.440806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.440773 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:18:05.571325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\") pod \"695563f5-e441-4538-b5b2-5617da6965b0\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " Oct 30 22:18:05.571325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571234 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\") pod \"f1a57585-851e-4269-83b1-23b7b478634d\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " Oct 30 22:18:05.571596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571364 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\") pod \"695563f5-e441-4538-b5b2-5617da6965b0\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " Oct 30 22:18:05.571596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571435 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\") pod \"f1a57585-851e-4269-83b1-23b7b478634d\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " Oct 30 22:18:05.571596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571494 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\") pod \"f1a57585-851e-4269-83b1-23b7b478634d\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " Oct 30 22:18:05.571596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571519 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\") pod \"695563f5-e441-4538-b5b2-5617da6965b0\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " Oct 30 22:18:05.571596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571555 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\") pod \"695563f5-e441-4538-b5b2-5617da6965b0\" (UID: \"695563f5-e441-4538-b5b2-5617da6965b0\") " Oct 30 22:18:05.571877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571634 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume" pod "695563f5-e441-4538-b5b2-5617da6965b0" (UID: "695563f5-e441-4538-b5b2-5617da6965b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:05.571984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume" pod "f1a57585-851e-4269-83b1-23b7b478634d" (UID: "f1a57585-851e-4269-83b1-23b7b478634d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:05.572082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.571979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\") pod \"f1a57585-851e-4269-83b1-23b7b478634d\" (UID: \"f1a57585-851e-4269-83b1-23b7b478634d\") " Oct 30 22:18:05.572082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.572013 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume" pod "695563f5-e441-4538-b5b2-5617da6965b0" (UID: "695563f5-e441-4538-b5b2-5617da6965b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:05.572498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.572465 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume" pod "f1a57585-851e-4269-83b1-23b7b478634d" (UID: "f1a57585-851e-4269-83b1-23b7b478634d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:05.573058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.573033 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.573118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.573067 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.573118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.573084 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a57585-851e-4269-83b1-23b7b478634d-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.573118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.573100 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695563f5-e441-4538-b5b2-5617da6965b0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.574165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.574144 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials" pod "f1a57585-851e-4269-83b1-23b7b478634d" (UID: "f1a57585-851e-4269-83b1-23b7b478634d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:05.574352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.574332 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt" pod "695563f5-e441-4538-b5b2-5617da6965b0" (UID: "695563f5-e441-4538-b5b2-5617da6965b0"). InnerVolumeSpecName "kube-api-access-tbllt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:05.574414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.574379 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials" pod "695563f5-e441-4538-b5b2-5617da6965b0" (UID: "695563f5-e441-4538-b5b2-5617da6965b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:05.574500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.574480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2" pod "f1a57585-851e-4269-83b1-23b7b478634d" (UID: "f1a57585-851e-4269-83b1-23b7b478634d"). InnerVolumeSpecName "kube-api-access-m7zv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:05.674391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.674362 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.674391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.674389 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbllt\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-kube-api-access-tbllt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.674573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.674400 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f1a57585-851e-4269-83b1-23b7b478634d-kube-api-access-m7zv2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.674573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.674412 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/695563f5-e441-4538-b5b2-5617da6965b0-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:05.890702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:05.890661 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" podUID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" containerName="payload-processing" containerID="cri-o://b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77" gracePeriod=25 Oct 30 22:18:06.212838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.212811 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:18:06.285575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.285538 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\") pod \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " Oct 30 22:18:06.285756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.285671 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\") pod \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " Oct 30 22:18:06.285756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.285702 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\") pod \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " Oct 30 22:18:06.285897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.285762 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\") pod \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\" (UID: \"9d448d61-91ae-47ab-a1c4-adcd968b3da2\") " Oct 30 22:18:06.285996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.285964 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume" pod "9d448d61-91ae-47ab-a1c4-adcd968b3da2" (UID: "9d448d61-91ae-47ab-a1c4-adcd968b3da2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:06.286067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.286048 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume" pod "9d448d61-91ae-47ab-a1c4-adcd968b3da2" (UID: "9d448d61-91ae-47ab-a1c4-adcd968b3da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:06.286717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.286698 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.286820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.286719 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d448d61-91ae-47ab-a1c4-adcd968b3da2-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.288159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.288136 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials" pod "9d448d61-91ae-47ab-a1c4-adcd968b3da2" (UID: "9d448d61-91ae-47ab-a1c4-adcd968b3da2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:06.288334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.288255 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd" pod "9d448d61-91ae-47ab-a1c4-adcd968b3da2" (UID: "9d448d61-91ae-47ab-a1c4-adcd968b3da2"). InnerVolumeSpecName "kube-api-access-jwppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:06.356063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.356034 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" Oct 30 22:18:06.356246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.356035 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8" event={"ID":"695563f5-e441-4538-b5b2-5617da6965b0","Type":"ContainerDied","Data":"ca2c0702dab4a003949840707111d271cbc8429d902a71331d06c1ef1c34b974"} Oct 30 22:18:06.356246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.356190 5036 scope.go:122] "RemoveContainer" containerID="f2baa6bf5f7fa5c40a08cf54caf8de09724e806523e1f95de0a5579d01185716" Oct 30 22:18:06.357848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.357822 5036 generic.go:356] "Generic (PLEG): container finished" podID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" containerID="b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77" exitCode=0 Oct 30 22:18:06.357999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.357986 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" Oct 30 22:18:06.359032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.358760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" event={"ID":"9d448d61-91ae-47ab-a1c4-adcd968b3da2","Type":"ContainerDied","Data":"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77"} Oct 30 22:18:06.359032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.358811 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg" event={"ID":"9d448d61-91ae-47ab-a1c4-adcd968b3da2","Type":"ContainerDied","Data":"c3fa2b18285c1adecafb05199471ff84b204514f2fb1cc158934cf7ceafe6fd2"} Oct 30 22:18:06.361035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.361010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" event={"ID":"f1a57585-851e-4269-83b1-23b7b478634d","Type":"ContainerDied","Data":"06fb208ee5c921dffec2a4a420c1dc5035ae722432e4a6b519416ddead1d7f0d"} Oct 30 22:18:06.361154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.361102 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz" Oct 30 22:18:06.371557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.371534 5036 scope.go:122] "RemoveContainer" containerID="b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77" Oct 30 22:18:06.384099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.384076 5036 scope.go:122] "RemoveContainer" containerID="b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77" Oct 30 22:18:06.384509 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:06.384476 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77\": container with ID starting with b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77 not found: ID does not exist" containerID="b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77" Oct 30 22:18:06.384643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.384514 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:18:06.384643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.384520 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77"} err="failed to get container status \"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77\": rpc error: code = NotFound desc = could not find container \"b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77\": container with ID starting with b7439fff1060ab9465016bd6822d9f5006959f7f777fd46c00e85ef329b96b77 not found: ID does not exist" Oct 30 22:18:06.384643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.384545 5036 scope.go:122] "RemoveContainer" containerID="c782e1cd76ddbc9d2f0511793498017e12e745e9f8742083523eee5f1115b7a1" Oct 30 22:18:06.388516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.388456 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwppd\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-kube-api-access-jwppd\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.388645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.388521 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d448d61-91ae-47ab-a1c4-adcd968b3da2-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.391489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.391464 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-845f2l8p8"] Oct 30 22:18:06.405757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.405736 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:18:06.427885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.427808 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbddp6vg"] Oct 30 22:18:06.429010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.428980 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:18:06.437849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:06.437821 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-6dd8n7rfz"] Oct 30 22:18:07.010955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:07.010920 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:18:07.084109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:07.084076 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:18:08.195236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.195201 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695563f5-e441-4538-b5b2-5617da6965b0" path="/var/lib/kubelet/pods/695563f5-e441-4538-b5b2-5617da6965b0/volumes" Oct 30 22:18:08.196033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.196014 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" path="/var/lib/kubelet/pods/9d448d61-91ae-47ab-a1c4-adcd968b3da2/volumes" Oct 30 22:18:08.196728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.196707 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a57585-851e-4269-83b1-23b7b478634d" path="/var/lib/kubelet/pods/f1a57585-851e-4269-83b1-23b7b478634d/volumes" Oct 30 22:18:08.313529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.313484 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" podUID="d0eea665-1822-43ca-9034-2da89c63a415" containerName="payload-processing" containerID="cri-o://1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1" gracePeriod=25 Oct 30 22:18:08.624625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.624598 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:08.773763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.773678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\") pod \"d0eea665-1822-43ca-9034-2da89c63a415\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " Oct 30 22:18:08.773946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.773833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\") pod \"d0eea665-1822-43ca-9034-2da89c63a415\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " Oct 30 22:18:08.773946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.773881 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\") pod \"d0eea665-1822-43ca-9034-2da89c63a415\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " Oct 30 22:18:08.774091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.773947 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\") pod \"d0eea665-1822-43ca-9034-2da89c63a415\" (UID: \"d0eea665-1822-43ca-9034-2da89c63a415\") " Oct 30 22:18:08.774150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.774108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume" pod "d0eea665-1822-43ca-9034-2da89c63a415" (UID: "d0eea665-1822-43ca-9034-2da89c63a415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:08.774347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.774320 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume" pod "d0eea665-1822-43ca-9034-2da89c63a415" (UID: "d0eea665-1822-43ca-9034-2da89c63a415"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:08.774639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.774622 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:08.774693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.774642 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0eea665-1822-43ca-9034-2da89c63a415-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:08.776177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.776153 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials" pod "d0eea665-1822-43ca-9034-2da89c63a415" (UID: "d0eea665-1822-43ca-9034-2da89c63a415"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:08.776256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.776177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc" pod "d0eea665-1822-43ca-9034-2da89c63a415" (UID: "d0eea665-1822-43ca-9034-2da89c63a415"). InnerVolumeSpecName "kube-api-access-wjphc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:08.877326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.877243 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:08.877326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:08.877299 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjphc\" (UniqueName: \"kubernetes.io/projected/d0eea665-1822-43ca-9034-2da89c63a415-kube-api-access-wjphc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.399755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.399719 5036 generic.go:356] "Generic (PLEG): container finished" podID="d0eea665-1822-43ca-9034-2da89c63a415" containerID="1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1" exitCode=0 Oct 30 22:18:09.400247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.399769 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" Oct 30 22:18:09.400247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.399800 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" event={"ID":"d0eea665-1822-43ca-9034-2da89c63a415","Type":"ContainerDied","Data":"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1"} Oct 30 22:18:09.400247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.399854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x" event={"ID":"d0eea665-1822-43ca-9034-2da89c63a415","Type":"ContainerDied","Data":"7e7e0adb2e238ab43b02c5f7b294e0c79969b6870455b229d924277a162a479d"} Oct 30 22:18:09.400247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.399884 5036 scope.go:122] "RemoveContainer" containerID="1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1" Oct 30 22:18:09.415172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.415148 5036 scope.go:122] "RemoveContainer" containerID="1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1" Oct 30 22:18:09.415528 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:09.415509 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1\": container with ID starting with 1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1 not found: ID does not exist" containerID="1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1" Oct 30 22:18:09.415612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.415537 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1"} err="failed to get container status \"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1\": rpc error: code = NotFound desc = could not find container \"1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1\": container with ID starting with 1a648d1b90d6003ffeddd4a731a11e6c285ac5663f18f95e11e193c9a894dce1 not found: ID does not exist" Oct 30 22:18:09.430105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.430080 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:09.435414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:09.435391 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-5f9czjh4x"] Oct 30 22:18:10.199001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:10.198964 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0eea665-1822-43ca-9034-2da89c63a415" path="/var/lib/kubelet/pods/d0eea665-1822-43ca-9034-2da89c63a415/volumes" Oct 30 22:18:12.085323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.085243 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" podUID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" containerName="payload-processing" containerID="cri-o://e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75" gracePeriod=25 Oct 30 22:18:12.400398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.400369 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:18:12.438377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.438314 5036 generic.go:356] "Generic (PLEG): container finished" podID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" containerID="e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75" exitCode=0 Oct 30 22:18:12.438573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.438369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" event={"ID":"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4","Type":"ContainerDied","Data":"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75"} Oct 30 22:18:12.438573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.438394 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" Oct 30 22:18:12.438573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.438462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk" event={"ID":"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4","Type":"ContainerDied","Data":"7aa37f71f3f69b7c4eebef4b48177225c438354d6f9ecdf5db3c29a990be0caa"} Oct 30 22:18:12.438573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.438513 5036 scope.go:122] "RemoveContainer" containerID="e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75" Oct 30 22:18:12.463942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.463830 5036 scope.go:122] "RemoveContainer" containerID="e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75" Oct 30 22:18:12.465007 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:12.464974 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75\": container with ID starting with e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75 not found: ID does not exist" containerID="e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75" Oct 30 22:18:12.465160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.465018 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75"} err="failed to get container status \"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75\": rpc error: code = NotFound desc = could not find container \"e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75\": container with ID starting with e36698125cb6142f598ab88daa4198d70f06d53d5ad90166852f9f2bac11ea75 not found: ID does not exist" Oct 30 22:18:12.557232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.557195 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\") pod \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " Oct 30 22:18:12.557568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.557547 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\") pod \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " Oct 30 22:18:12.557702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.557677 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\") pod \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " Oct 30 22:18:12.557896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.557732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\") pod \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\" (UID: \"1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4\") " Oct 30 22:18:12.558099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.557979 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume" pod "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" (UID: "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:12.558370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.558347 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume" pod "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" (UID: "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:12.558634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.558609 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:12.558717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.558640 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:12.559894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.559869 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq" pod "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" (UID: "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4"). InnerVolumeSpecName "kube-api-access-gbkkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:12.560738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.560713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials" pod "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" (UID: "1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:12.660629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.660545 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkkq\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-kube-api-access-gbkkq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:12.660629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.660590 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:12.769123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.769093 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:18:12.774436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:12.774408 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cbb97f857-xv7kk"] Oct 30 22:18:14.191412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:14.191383 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" path="/var/lib/kubelet/pods/1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4/volumes" Oct 30 22:18:18.103633 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:18.103587 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:18:19.167336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.165890 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168306 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168331 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168362 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168370 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168389 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d0eea665-1822-43ca-9034-2da89c63a415" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168397 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0eea665-1822-43ca-9034-2da89c63a415" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168430 5036 cpu_manager.go:414] "containerMap: removing container" podUID="695563f5-e441-4538-b5b2-5617da6965b0" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168437 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695563f5-e441-4538-b5b2-5617da6965b0" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168479 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f1a57585-851e-4269-83b1-23b7b478634d" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168488 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a57585-851e-4269-83b1-23b7b478634d" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168682 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="695563f5-e441-4538-b5b2-5617da6965b0" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168704 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a57585-851e-4269-83b1-23b7b478634d" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168728 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8e6cfe-6612-415e-ae4d-b6d3bc29dfc4" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168739 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d448d61-91ae-47ab-a1c4-adcd968b3da2" containerName="payload-processing" Oct 30 22:18:19.170012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.168749 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0eea665-1822-43ca-9034-2da89c63a415" containerName="payload-processing" Oct 30 22:18:19.180702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.179154 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.191288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.188928 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-67105abf9904\"" Oct 30 22:18:19.191288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.189506 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:19.261797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.261756 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.262068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.262048 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6lqd\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.262249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.262231 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.262475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.262455 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.364402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.364365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.365725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.365702 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6lqd\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.366242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.366219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.367082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.367060 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.367349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.366981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.367964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.365641 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.379666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.379647 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.380873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.380854 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6lqd\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\") pod \"payload-processing-external-model-5685d65658-b52bp\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.502510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.502428 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:19.894195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:19.894169 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:20.537640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.537549 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:20.570318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.565407 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.580592 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.582663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.589649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.589679 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.589983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgck7\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.590499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.588337 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:20.633290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.631167 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3acda20c8a34\"" Oct 30 22:18:20.691397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.691363 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.691558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.691401 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.691558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.691422 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.691654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.691637 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgck7\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.692159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.692091 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.692159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.692098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.694256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.694233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.704587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.704567 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgck7\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\") pod \"payload-processing-external-model-6dd446f68c-qbtkn\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:20.717837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.717351 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" event={"ID":"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac","Type":"ContainerStarted","Data":"0389600042f8f323c0c2bdef51f82326b86696ac1ccd99934b091b796acc141c"} Oct 30 22:18:20.717837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.717413 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" event={"ID":"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac","Type":"ContainerStarted","Data":"45bb1e1cd46429d3ed5366ee90647903481665f1856f58da6674e6e61ea36d94"} Oct 30 22:18:20.718505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.718483 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:20.941411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:20.941377 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:21.257090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.256962 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" podStartSLOduration=2.256941212 podStartE2EDuration="2.256941212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:20.765297245 +0000 UTC m=+2324.715252194" watchObservedRunningTime="2026-10-30 22:18:21.256941212 +0000 UTC m=+2325.206896159" Oct 30 22:18:21.262533 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:18:21.262499 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba203432_a051_4c0d_ab06_36407718aa71.slice/crio-f19d4f786cef4eed6ca09cb07b6278e6473376bbeb29555541eb304f4b05c013 WatchSource:0}: Error finding container f19d4f786cef4eed6ca09cb07b6278e6473376bbeb29555541eb304f4b05c013: Status 404 returned error can't find the container with id f19d4f786cef4eed6ca09cb07b6278e6473376bbeb29555541eb304f4b05c013 Oct 30 22:18:21.262663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.262646 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:21.731322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.731280 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" event={"ID":"ba203432-a051-4c0d-ab06-36407718aa71","Type":"ContainerStarted","Data":"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a"} Oct 30 22:18:21.731846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.731819 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:21.731974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.731849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" event={"ID":"ba203432-a051-4c0d-ab06-36407718aa71","Type":"ContainerStarted","Data":"f19d4f786cef4eed6ca09cb07b6278e6473376bbeb29555541eb304f4b05c013"} Oct 30 22:18:21.782005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:21.781958 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" podStartSLOduration=1.781944324 podStartE2EDuration="1.781944324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:21.778897701 +0000 UTC m=+2325.728852646" watchObservedRunningTime="2026-10-30 22:18:21.781944324 +0000 UTC m=+2325.731899276" Oct 30 22:18:25.163125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.152889 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:25.243574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.243535 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:18:25.266209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.266178 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.318140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.318099 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.318873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.318768 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.320397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.320365 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bmj\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.321854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.321834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.328215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.328192 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:18:25.425321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.424221 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bmj\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.425321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.424371 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.425321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.424561 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.425321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.424644 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.425655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.425472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.426219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.426190 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.428913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.428434 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.452802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.452780 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bmj\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\") pod \"payload-processing-external-model-5685d65658-dqm8t\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:25.672296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:25.671561 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:26.249390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.249339 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:18:26.731885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.731835 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" podUID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" containerName="payload-processing" containerID="cri-o://0389600042f8f323c0c2bdef51f82326b86696ac1ccd99934b091b796acc141c" gracePeriod=25 Oct 30 22:18:26.870691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.870655 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" event={"ID":"2ff96c79-dd9b-434a-8ad2-80880f8c2530","Type":"ContainerStarted","Data":"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81"} Oct 30 22:18:26.870863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.870717 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" event={"ID":"2ff96c79-dd9b-434a-8ad2-80880f8c2530","Type":"ContainerStarted","Data":"d46ca8113aa960ea9b61e1e1c670145dbcc42507a73db34c24d29c32e6456e9d"} Oct 30 22:18:26.870863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.870746 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:26.872610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.872585 5036 generic.go:356] "Generic (PLEG): container finished" podID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" containerID="0389600042f8f323c0c2bdef51f82326b86696ac1ccd99934b091b796acc141c" exitCode=0 Oct 30 22:18:26.872749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.872656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" event={"ID":"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac","Type":"ContainerDied","Data":"0389600042f8f323c0c2bdef51f82326b86696ac1ccd99934b091b796acc141c"} Oct 30 22:18:26.896112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:26.896065 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" podStartSLOduration=1.896052227 podStartE2EDuration="1.896052227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:26.892641404 +0000 UTC m=+2330.842596349" watchObservedRunningTime="2026-10-30 22:18:26.896052227 +0000 UTC m=+2330.846007194" Oct 30 22:18:27.032333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.032304 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:27.054666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.054632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\") pod \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.054857 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\") pod \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.054900 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\") pod \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.055048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\") pod \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\" (UID: \"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac\") " Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.055091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume" pod "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" (UID: "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.055251 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume" pod "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" (UID: "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.055818 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.055894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.055840 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.058435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.057988 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials" pod "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" (UID: "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:27.058435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.058389 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd" pod "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" (UID: "21b4b5cc-7d02-4c07-95e7-afde9a22a0ac"). InnerVolumeSpecName "kube-api-access-l6lqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:27.157978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.157938 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6lqd\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-kube-api-access-l6lqd\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.158132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.157970 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.440691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.440656 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:18:27.442028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.441998 5036 cpu_manager.go:414] "containerMap: removing container" podUID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" containerName="payload-processing" Oct 30 22:18:27.442028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.442027 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" containerName="payload-processing" Oct 30 22:18:27.442323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.442308 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" containerName="payload-processing" Oct 30 22:18:27.447397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.447377 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.463081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463052 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.463200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.463441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463413 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.463538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463485 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.463601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.464787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463720 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rgg\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.464787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463780 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.464787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463821 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.464787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.463942 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.472182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.472159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:18:27.565770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.565941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.565941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565817 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.565941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565860 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rgg\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.565941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565896 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.565941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565920 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.565964 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566049 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566453 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566532 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.566667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.566648 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.567068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.567044 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.568301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.568278 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.568756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.568733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.579355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.579330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.580122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.580101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rgg\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-w7g8z\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.777783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.777693 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:27.904118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.904078 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" Oct 30 22:18:27.904332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.904139 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-b52bp" event={"ID":"21b4b5cc-7d02-4c07-95e7-afde9a22a0ac","Type":"ContainerDied","Data":"45bb1e1cd46429d3ed5366ee90647903481665f1856f58da6674e6e61ea36d94"} Oct 30 22:18:27.904332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.904227 5036 scope.go:122] "RemoveContainer" containerID="0389600042f8f323c0c2bdef51f82326b86696ac1ccd99934b091b796acc141c" Oct 30 22:18:27.954318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.953326 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:27.961825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:27.961802 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-b52bp"] Oct 30 22:18:28.107214 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:18:28.107169 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2dfe1f_6599_40a0_9369_f7f720c90647.slice/crio-ad2adf394a4168cd6a43f16ba4185d382bda24e1ee13318f54dfd39d91c9aa37 WatchSource:0}: Error finding container ad2adf394a4168cd6a43f16ba4185d382bda24e1ee13318f54dfd39d91c9aa37: Status 404 returned error can't find the container with id ad2adf394a4168cd6a43f16ba4185d382bda24e1ee13318f54dfd39d91c9aa37 Oct 30 22:18:28.109126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.108999 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:18:28.110433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.110395 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:28.110545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.110462 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:28.110545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.110492 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:28.192844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.192820 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b4b5cc-7d02-4c07-95e7-afde9a22a0ac" path="/var/lib/kubelet/pods/21b4b5cc-7d02-4c07-95e7-afde9a22a0ac/volumes" Oct 30 22:18:28.930930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.930887 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" event={"ID":"8c2dfe1f-6599-40a0-9369-f7f720c90647","Type":"ContainerStarted","Data":"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94"} Oct 30 22:18:28.931593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:28.931563 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" event={"ID":"8c2dfe1f-6599-40a0-9369-f7f720c90647","Type":"ContainerStarted","Data":"ad2adf394a4168cd6a43f16ba4185d382bda24e1ee13318f54dfd39d91c9aa37"} Oct 30 22:18:29.053055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:29.052966 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podStartSLOduration=2.05294753 podStartE2EDuration="2.05294753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.050605058 +0000 UTC m=+2333.000560006" watchObservedRunningTime="2026-10-30 22:18:29.05294753 +0000 UTC m=+2333.002902477" Oct 30 22:18:29.778752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:29.778707 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:29.780762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:29.780733 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:29.781037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:29.780981 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:30.050324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.050189 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:18:30.056992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.056966 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.059337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.058439 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.060885 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-e2e--2051f056-dockercfg-d4jzz\"" Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.060932 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-fb8dfd3b6ae8\"" Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.061766 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"kube-root-ca.crt\"" Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.062412 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"routing-overlay\"" Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.062477 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9d868\"" Oct 30 22:18:30.063334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.062556 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"openshift-service-ca.crt\"" Oct 30 22:18:30.160298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.156648 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" podUID="ba203432-a051-4c0d-ab06-36407718aa71" containerName="payload-processing" containerID="cri-o://e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a" gracePeriod=25 Oct 30 22:18:30.264296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.260523 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.264296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.260585 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.264296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.260793 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.264296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.260865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2b5\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.368346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.368229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.368346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.368326 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.368592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.368557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.368648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.368635 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2b5\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.372323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.369991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.372323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.370676 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.440291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.413100 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.440291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.428909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2b5\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.710737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.710207 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:30.778925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.778451 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:30.778925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.778512 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:30.867954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.867915 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:30.983404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.981208 5036 generic.go:356] "Generic (PLEG): container finished" podID="ba203432-a051-4c0d-ab06-36407718aa71" containerID="e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a" exitCode=0 Oct 30 22:18:30.983404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.982060 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" Oct 30 22:18:30.983404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.982566 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" event={"ID":"ba203432-a051-4c0d-ab06-36407718aa71","Type":"ContainerDied","Data":"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a"} Oct 30 22:18:30.983404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.982620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn" event={"ID":"ba203432-a051-4c0d-ab06-36407718aa71","Type":"ContainerDied","Data":"f19d4f786cef4eed6ca09cb07b6278e6473376bbeb29555541eb304f4b05c013"} Oct 30 22:18:30.983404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.982646 5036 scope.go:122] "RemoveContainer" containerID="e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a" Oct 30 22:18:30.999346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:30.999318 5036 scope.go:122] "RemoveContainer" containerID="e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a" Oct 30 22:18:31.001516 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:31.001488 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a\": container with ID starting with e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a not found: ID does not exist" containerID="e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a" Oct 30 22:18:31.001653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.001528 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a"} err="failed to get container status \"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a\": rpc error: code = NotFound desc = could not find container \"e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a\": container with ID starting with e405b6e92eae6ded19a239861fc7c28ab174c4ae95fabe4eb16d5d6bb65ba54a not found: ID does not exist" Oct 30 22:18:31.057473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.057417 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\") pod \"ba203432-a051-4c0d-ab06-36407718aa71\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " Oct 30 22:18:31.057944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.057667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\") pod \"ba203432-a051-4c0d-ab06-36407718aa71\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " Oct 30 22:18:31.058038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.057976 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\") pod \"ba203432-a051-4c0d-ab06-36407718aa71\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " Oct 30 22:18:31.058095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.058033 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\") pod \"ba203432-a051-4c0d-ab06-36407718aa71\" (UID: \"ba203432-a051-4c0d-ab06-36407718aa71\") " Oct 30 22:18:31.058722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.058687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume" pod "ba203432-a051-4c0d-ab06-36407718aa71" (UID: "ba203432-a051-4c0d-ab06-36407718aa71"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:31.059536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.059506 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.077362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.074919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7" pod "ba203432-a051-4c0d-ab06-36407718aa71" (UID: "ba203432-a051-4c0d-ab06-36407718aa71"). InnerVolumeSpecName "kube-api-access-pgck7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:31.077362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.075965 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials" pod "ba203432-a051-4c0d-ab06-36407718aa71" (UID: "ba203432-a051-4c0d-ab06-36407718aa71"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:31.079120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.079093 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume" pod "ba203432-a051-4c0d-ab06-36407718aa71" (UID: "ba203432-a051-4c0d-ab06-36407718aa71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:31.147371 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:18:31.147334 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13718ae8_7811_45d7_9698_1978db6e8498.slice/crio-03dc43baae7c61dec1042d05891249ac467f17957741fd383188229374cd35db WatchSource:0}: Error finding container 03dc43baae7c61dec1042d05891249ac467f17957741fd383188229374cd35db: Status 404 returned error can't find the container with id 03dc43baae7c61dec1042d05891249ac467f17957741fd383188229374cd35db Oct 30 22:18:31.152733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.152703 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:18:31.161965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.160843 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.162102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.161980 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba203432-a051-4c0d-ab06-36407718aa71-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.162102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.162003 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgck7\" (UniqueName: \"kubernetes.io/projected/ba203432-a051-4c0d-ab06-36407718aa71-kube-api-access-pgck7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.356603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.356560 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:31.365839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.365808 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dd446f68c-qbtkn"] Oct 30 22:18:31.778752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.778710 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:31.778961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.778777 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:31.994800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.994760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" event={"ID":"13718ae8-7811-45d7-9698-1978db6e8498","Type":"ContainerStarted","Data":"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813"} Oct 30 22:18:31.994988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.994821 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" event={"ID":"13718ae8-7811-45d7-9698-1978db6e8498","Type":"ContainerStarted","Data":"03dc43baae7c61dec1042d05891249ac467f17957741fd383188229374cd35db"} Oct 30 22:18:31.995173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:31.994987 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:32.032350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:32.029891 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" podStartSLOduration=2.029871355 podStartE2EDuration="2.029871355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:32.029305819 +0000 UTC m=+2335.979260765" watchObservedRunningTime="2026-10-30 22:18:32.029871355 +0000 UTC m=+2335.979826302" Oct 30 22:18:32.190846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:32.190814 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba203432-a051-4c0d-ab06-36407718aa71" path="/var/lib/kubelet/pods/ba203432-a051-4c0d-ab06-36407718aa71/volumes" Oct 30 22:18:32.778809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:32.778769 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:32.779019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:32.778830 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:33.778165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:33.778125 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:33.778795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:33.778192 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:34.778523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:34.778487 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:34.778950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:34.778541 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:35.785879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:35.785844 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-w7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Oct 30 22:18:35.786566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:35.786531 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Oct 30 22:18:37.618418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:37.616895 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:37.619994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:37.619971 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:37.620662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:37.620641 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:18:37.907916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:37.907887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:18:37.972179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:37.972141 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:18:39.991276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:39.991240 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 22:18:39.991806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:39.991472 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" containerID="cri-o://21fdc22a7972093f5f31d43218a897f67d1112b17fdcfc46d5c710a93ba5186f" gracePeriod=30 Oct 30 22:18:42.481706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.481668 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl"] Oct 30 22:18:42.483046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.483015 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ba203432-a051-4c0d-ab06-36407718aa71" containerName="payload-processing" Oct 30 22:18:42.483215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.483050 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba203432-a051-4c0d-ab06-36407718aa71" containerName="payload-processing" Oct 30 22:18:42.483459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.483415 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba203432-a051-4c0d-ab06-36407718aa71" containerName="payload-processing" Oct 30 22:18:42.490937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.490912 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.503914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.503880 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl"] Oct 30 22:18:42.516584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.516550 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.516751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.516612 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.516820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.516785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.516955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.516910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.517019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.517002 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.517071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.517035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fts\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-kube-api-access-29fts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.517071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.517062 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.517318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.517246 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.517378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.517319 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.618611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618574 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.618810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618616 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fts\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-kube-api-access-29fts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.618810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618640 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.618810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618719 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.618810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618769 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.618940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619119 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619130 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619359 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619505 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619600 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.619891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.619827 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.621460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.621439 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.621571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.621551 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.630325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.630298 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.630578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.630561 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fts\" (UniqueName: \"kubernetes.io/projected/d58ccfb1-e100-4d6f-9974-17dab6bbd846-kube-api-access-29fts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c7wkl\" (UID: \"d58ccfb1-e100-4d6f-9974-17dab6bbd846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.819508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.819429 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:42.972865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:42.972814 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" podUID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" containerName="payload-processing" containerID="cri-o://227b8da3b3769efcdff6518de24a60cd3f7c99906a3c6714315a176d68a10b59" gracePeriod=25 Oct 30 22:18:43.058793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.058763 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:18:43.143351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.143061 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl"] Oct 30 22:18:43.146134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.146094 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:43.146300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.146162 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:43.146300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.146197 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:43.312765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.312728 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" event={"ID":"d58ccfb1-e100-4d6f-9974-17dab6bbd846","Type":"ContainerStarted","Data":"2017d8dc5c3363d94e3d724ba46d7d784a095b3bb3b904c14c0540291bff2c9b"} Oct 30 22:18:43.314773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.314749 5036 generic.go:356] "Generic (PLEG): container finished" podID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" containerID="227b8da3b3769efcdff6518de24a60cd3f7c99906a3c6714315a176d68a10b59" exitCode=0 Oct 30 22:18:43.315054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.315040 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" event={"ID":"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3","Type":"ContainerDied","Data":"227b8da3b3769efcdff6518de24a60cd3f7c99906a3c6714315a176d68a10b59"} Oct 30 22:18:43.352343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.351662 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:18:43.523230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.523197 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\") pod \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " Oct 30 22:18:43.523761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.523343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\") pod \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " Oct 30 22:18:43.523761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.523520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\") pod \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " Oct 30 22:18:43.523761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.523592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\") pod \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\" (UID: \"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3\") " Oct 30 22:18:43.523967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.523939 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume" pod "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" (UID: "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:43.524103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.524060 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume" pod "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" (UID: "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:43.524544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.524487 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:43.524544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.524529 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:43.526086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.526062 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp" pod "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" (UID: "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3"). InnerVolumeSpecName "kube-api-access-bsnlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:43.526193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.526113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials" pod "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" (UID: "2ffaa5ed-53a9-442f-b1ba-5386452ba2f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:43.625953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.625866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnlp\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-kube-api-access-bsnlp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:43.625953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:43.625895 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:44.353837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.353794 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" event={"ID":"d58ccfb1-e100-4d6f-9974-17dab6bbd846","Type":"ContainerStarted","Data":"823a622168f431a6258a8153d9d2c710f94d660055e7596faf5b8eb627b3db69"} Oct 30 22:18:44.356567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.356543 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" Oct 30 22:18:44.356857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.356551 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk" event={"ID":"2ffaa5ed-53a9-442f-b1ba-5386452ba2f3","Type":"ContainerDied","Data":"2980a8871e0a76a2a0e78e5e00f3c60735492c70703640c16186ee73cb7a5c78"} Oct 30 22:18:44.356947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.356891 5036 scope.go:122] "RemoveContainer" containerID="227b8da3b3769efcdff6518de24a60cd3f7c99906a3c6714315a176d68a10b59" Oct 30 22:18:44.377733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.377692 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podStartSLOduration=2.377679598 podStartE2EDuration="2.377679598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:44.376116931 +0000 UTC m=+2348.326071874" watchObservedRunningTime="2026-10-30 22:18:44.377679598 +0000 UTC m=+2348.327634543" Oct 30 22:18:44.398041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.398019 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:18:44.405366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.405343 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84ff676cfc-svfjk"] Oct 30 22:18:44.821707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.821666 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:44.838288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.833393 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.838288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:44.833456 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:45.375719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.375687 5036 generic.go:356] "Generic (PLEG): container finished" podID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerID="21fdc22a7972093f5f31d43218a897f67d1112b17fdcfc46d5c710a93ba5186f" exitCode=0 Oct 30 22:18:45.375959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.375718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" event={"ID":"47dcda32-bf9b-4260-875e-0eceacf7be36","Type":"ContainerDied","Data":"21fdc22a7972093f5f31d43218a897f67d1112b17fdcfc46d5c710a93ba5186f"} Oct 30 22:18:45.376043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.375991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" event={"ID":"47dcda32-bf9b-4260-875e-0eceacf7be36","Type":"ContainerDied","Data":"45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f"} Oct 30 22:18:45.376043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.376008 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45f6572a87abfad20639f515dae95f5247ae5fa53e89897661f81901d21bee2f" Oct 30 22:18:45.395518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.395483 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 22:18:45.590667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.590637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.591056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.591031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.591291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.591246 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.591515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.591481 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.591976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.591958 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.592122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592109 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.592296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592257 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.592497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592466 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.592680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592651 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\") pod \"47dcda32-bf9b-4260-875e-0eceacf7be36\" (UID: \"47dcda32-bf9b-4260-875e-0eceacf7be36\") " Oct 30 22:18:45.592834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:45.592924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:45.592924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.592869 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:45.593140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.593114 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:45.593228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.593146 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:45.593884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.593862 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:45.594185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594170 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.594309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594297 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.594405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594396 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.594480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594472 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.594559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594551 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dcda32-bf9b-4260-875e-0eceacf7be36-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.594632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.594625 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.595395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.595335 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "kube-api-access-2p6qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:45.595743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.595688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:45.595849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.595828 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo" pod "47dcda32-bf9b-4260-875e-0eceacf7be36" (UID: "47dcda32-bf9b-4260-875e-0eceacf7be36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:45.696917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.696880 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.696917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.696916 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p6qz\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-kube-api-access-2p6qz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.697161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.696931 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dcda32-bf9b-4260-875e-0eceacf7be36-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:45.830293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.824639 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:45.830293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:45.824713 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:46.190788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.190754 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" path="/var/lib/kubelet/pods/2ffaa5ed-53a9-442f-b1ba-5386452ba2f3/volumes" Oct 30 22:18:46.406086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.406033 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk" Oct 30 22:18:46.464471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.464380 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 22:18:46.472644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.472616 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-zvlfk"] Oct 30 22:18:46.830348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.826664 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:46.830348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:46.826739 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:47.824289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:47.820634 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:47.824289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:47.820698 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:47.912332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:47.910535 5036 scope.go:122] "RemoveContainer" containerID="21fdc22a7972093f5f31d43218a897f67d1112b17fdcfc46d5c710a93ba5186f" Oct 30 22:18:47.955522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:47.955497 5036 scope.go:122] "RemoveContainer" containerID="33073f856f609d9e4a7a78f2e4ca6d1c05c3da3c19357371b689870f2f3ad7cd" Oct 30 22:18:48.255501 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:18:48.255460 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:18:48.263161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:48.263127 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" path="/var/lib/kubelet/pods/47dcda32-bf9b-4260-875e-0eceacf7be36/volumes" Oct 30 22:18:48.830290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:48.825099 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:48.830290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:48.825187 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:49.819968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:49.819928 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:49.820405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:49.820009 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:50.820176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:50.820134 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c7wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:50.820639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:50.820208 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" podUID="d58ccfb1-e100-4d6f-9974-17dab6bbd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 30 22:18:52.349294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:52.348624 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:52.353285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:52.350397 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:52.365293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:52.361423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c7wkl" Oct 30 22:18:58.857477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:58.857438 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:58.858160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:58.858127 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:59.087142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:59.087105 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr"] Oct 30 22:18:59.155688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:18:59.155480 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bece7096-dq6zr"] Oct 30 22:19:00.191177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:00.191147 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5508dbea-bc66-4991-a689-f1baaabd5e6a" path="/var/lib/kubelet/pods/5508dbea-bc66-4991-a689-f1baaabd5e6a/volumes" Oct 30 22:19:02.039441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:02.039408 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw"] Oct 30 22:19:02.048100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:02.048065 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e705b3-qjssw"] Oct 30 22:19:02.191364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:02.191332 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74a80ea-c6f9-4e55-8729-3f4c434ed52a" path="/var/lib/kubelet/pods/a74a80ea-c6f9-4e55-8729-3f4c434ed52a/volumes" Oct 30 22:19:10.027381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:10.027345 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq"] Oct 30 22:19:10.033364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:10.033332 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac08523-c9phq"] Oct 30 22:19:10.190523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:10.190483 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a210acb-cbd9-4134-be02-a1ce8bc48152" path="/var/lib/kubelet/pods/8a210acb-cbd9-4134-be02-a1ce8bc48152/volumes" Oct 30 22:19:13.035559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:13.035511 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7"] Oct 30 22:19:13.040884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:13.040853 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185c28b-pxdz7"] Oct 30 22:19:14.228155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:14.228117 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70506294-9e8f-4aee-9870-1961472ca651" path="/var/lib/kubelet/pods/70506294-9e8f-4aee-9870-1961472ca651/volumes" Oct 30 22:19:18.102249 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:19:18.102156 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:19:31.593799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.593758 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt"] Oct 30 22:19:31.594894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.594867 5036 cpu_manager.go:414] "containerMap: removing container" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" Oct 30 22:19:31.594894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.594893 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" Oct 30 22:19:31.595079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.594963 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" containerName="payload-processing" Oct 30 22:19:31.595079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.594971 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" containerName="payload-processing" Oct 30 22:19:31.595222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.595208 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffaa5ed-53a9-442f-b1ba-5386452ba2f3" containerName="payload-processing" Oct 30 22:19:31.595305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.595239 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dcda32-bf9b-4260-875e-0eceacf7be36" containerName="istio-proxy" Oct 30 22:19:31.599865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.599842 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.604527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.604488 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:31.605416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.605390 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:19:31.605634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.605591 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:31.605986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.605648 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:31.609543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.609520 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt"] Oct 30 22:19:31.724465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.724430 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.724679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.724478 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.724679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.724655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwtt4\" (UniqueName: \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.825922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.825889 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.825922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.825929 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.826205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.826055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwtt4\" (UniqueName: \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.826664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.826638 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.828805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.828778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.843178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.843156 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwtt4\" (UniqueName: \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\") pod \"maas-api-revoke-sub-65c6fec3-d49qt\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:31.920038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:31.920006 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:32.218685 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:19:32.218597 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4f9825_23ec_4c66_a47d_3f4b66e90c5a.slice/crio-13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf WatchSource:0}: Error finding container 13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf: Status 404 returned error can't find the container with id 13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf Oct 30 22:19:32.221147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:32.221118 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt"] Oct 30 22:19:32.302776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:32.302734 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" event={"ID":"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a","Type":"ContainerStarted","Data":"13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf"} Oct 30 22:19:33.314142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:33.314098 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" event={"ID":"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a","Type":"ContainerStarted","Data":"0a7e1ee8b06cea5ee854971619353cc0adc66acbda0728b09bce76f4baeee3a4"} Oct 30 22:19:33.341108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:33.341045 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" podStartSLOduration=2.341025569 podStartE2EDuration="2.341025569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:33.33073833 +0000 UTC m=+2397.280693279" watchObservedRunningTime="2026-10-30 22:19:33.341025569 +0000 UTC m=+2397.290980515" Oct 30 22:19:34.387677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:34.387640 5036 generic.go:356] "Generic (PLEG): container finished" podID="5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" containerID="0a7e1ee8b06cea5ee854971619353cc0adc66acbda0728b09bce76f4baeee3a4" exitCode=0 Oct 30 22:19:34.406382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:34.388493 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" event={"ID":"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a","Type":"ContainerDied","Data":"0a7e1ee8b06cea5ee854971619353cc0adc66acbda0728b09bce76f4baeee3a4"} Oct 30 22:19:35.604661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.604624 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:35.707430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.707338 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\") pod \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " Oct 30 22:19:35.707596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.707525 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\" (UniqueName: \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\") pod \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " Oct 30 22:19:35.707873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.707856 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\") pod \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\" (UID: \"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a\") " Oct 30 22:19:35.708239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.708214 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca" pod "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" (UID: "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:35.708716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.708692 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:35.710230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.710203 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4" pod "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" (UID: "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a"). InnerVolumeSpecName "kube-api-access-lwtt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:35.710383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.710310 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca" pod "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" (UID: "5e4f9825-23ec-4c66-a47d-3f4b66e90c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:35.829290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.824421 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:35.829290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:35.824472 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwtt4\" (UniqueName: \"kubernetes.io/projected/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a-kube-api-access-lwtt4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.409324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:36.409284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" event={"ID":"5e4f9825-23ec-4c66-a47d-3f4b66e90c5a","Type":"ContainerDied","Data":"13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf"} Oct 30 22:19:36.409530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:36.409347 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e4681c2e2c4b1518e7d65223d4f0335bbe41ab169d4b54e64d33b6d54ea1bf" Oct 30 22:19:36.409530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:36.409255 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt" Oct 30 22:19:37.132437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:37.132403 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:19:37.134360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:37.134333 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:19:42.784700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:42.784517 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:19:45.104414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.104370 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:45.469295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.457386 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:19:45.567444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.515734 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:19:45.567444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.517163 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" containerName="revoke-keys" Oct 30 22:19:45.567444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.517189 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" containerName="revoke-keys" Oct 30 22:19:45.567444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.517459 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" containerName="revoke-keys" Oct 30 22:19:45.685296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.610927 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:45.819620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.819539 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:19:45.819969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.819950 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.831990 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-e2e--fed5953e-dockercfg-9jvpx\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832080 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-e53124f008b8\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832141 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"openshift-service-ca.crt\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832228 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"kube-root-ca.crt\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832305 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a5f192\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832402 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"routing-overlay\"" Oct 30 22:19:45.841300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:45.832462 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:19:46.016080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.011169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.016689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.016656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.019507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.019479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.020897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.020867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjdh\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.174286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.163669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.174286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.171824 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.174286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.171986 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjdh\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.174286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.172028 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.175621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.165031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.175869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.175829 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.302290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.244542 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjdh\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.303154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.271797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:46.435640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:46.435544 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:47.488947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.488897 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:19:47.507403 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:19:47.507347 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fc6654_747d_43cf_a178_0919a87344b2.slice/crio-fcb59d8474a78c37986351494a0a3993b327684a3cfb8b762ece354a191b0f20 WatchSource:0}: Error finding container fcb59d8474a78c37986351494a0a3993b327684a3cfb8b762ece354a191b0f20: Status 404 returned error can't find the container with id fcb59d8474a78c37986351494a0a3993b327684a3cfb8b762ece354a191b0f20 Oct 30 22:19:47.784992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.784914 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" podUID="b7aac589-0369-4952-be8b-5ee693d6db8b" containerName="payload-processing" containerID="cri-o://48358710faddc71083977ce33b8753fb8127ed9997df8e078112cbaa53b98cc5" gracePeriod=25 Oct 30 22:19:47.928356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.927985 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" event={"ID":"90fc6654-747d-43cf-a178-0919a87344b2","Type":"ContainerStarted","Data":"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7"} Oct 30 22:19:47.928356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.928088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" event={"ID":"90fc6654-747d-43cf-a178-0919a87344b2","Type":"ContainerStarted","Data":"fcb59d8474a78c37986351494a0a3993b327684a3cfb8b762ece354a191b0f20"} Oct 30 22:19:47.928356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.928118 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:47.930389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.930362 5036 generic.go:356] "Generic (PLEG): container finished" podID="b7aac589-0369-4952-be8b-5ee693d6db8b" containerID="48358710faddc71083977ce33b8753fb8127ed9997df8e078112cbaa53b98cc5" exitCode=0 Oct 30 22:19:47.930520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.930447 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" event={"ID":"b7aac589-0369-4952-be8b-5ee693d6db8b","Type":"ContainerDied","Data":"48358710faddc71083977ce33b8753fb8127ed9997df8e078112cbaa53b98cc5"} Oct 30 22:19:47.952140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:47.952093 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" podStartSLOduration=2.952079195 podStartE2EDuration="2.952079195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:47.945952029 +0000 UTC m=+2411.895906973" watchObservedRunningTime="2026-10-30 22:19:47.952079195 +0000 UTC m=+2411.902034141" Oct 30 22:19:48.152335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.152307 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:19:48.168243 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:19:48.168195 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:19:48.338785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.338703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\") pod \"b7aac589-0369-4952-be8b-5ee693d6db8b\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " Oct 30 22:19:48.338785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.338742 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\") pod \"b7aac589-0369-4952-be8b-5ee693d6db8b\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " Oct 30 22:19:48.339039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.338881 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\") pod \"b7aac589-0369-4952-be8b-5ee693d6db8b\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " Oct 30 22:19:48.339039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.338963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\") pod \"b7aac589-0369-4952-be8b-5ee693d6db8b\" (UID: \"b7aac589-0369-4952-be8b-5ee693d6db8b\") " Oct 30 22:19:48.339210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.339099 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume" pod "b7aac589-0369-4952-be8b-5ee693d6db8b" (UID: "b7aac589-0369-4952-be8b-5ee693d6db8b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:48.339345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.339322 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume" pod "b7aac589-0369-4952-be8b-5ee693d6db8b" (UID: "b7aac589-0369-4952-be8b-5ee693d6db8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:48.340272 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.340243 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:48.340414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.340356 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aac589-0369-4952-be8b-5ee693d6db8b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:48.341607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.341581 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials" pod "b7aac589-0369-4952-be8b-5ee693d6db8b" (UID: "b7aac589-0369-4952-be8b-5ee693d6db8b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:48.341727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.341706 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc" pod "b7aac589-0369-4952-be8b-5ee693d6db8b" (UID: "b7aac589-0369-4952-be8b-5ee693d6db8b"). InnerVolumeSpecName "kube-api-access-8fjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:48.442946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.442902 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:48.443242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.443219 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjhc\" (UniqueName: \"kubernetes.io/projected/b7aac589-0369-4952-be8b-5ee693d6db8b-kube-api-access-8fjhc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:48.941785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.941753 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" Oct 30 22:19:48.942312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.941783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7" event={"ID":"b7aac589-0369-4952-be8b-5ee693d6db8b","Type":"ContainerDied","Data":"2b230578f822ac5b3c2c56d0ad06c758e912985fd2ad5c24152604a1d4230a8c"} Oct 30 22:19:48.942312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.941855 5036 scope.go:122] "RemoveContainer" containerID="48358710faddc71083977ce33b8753fb8127ed9997df8e078112cbaa53b98cc5" Oct 30 22:19:48.978299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.978250 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:19:48.984484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:48.984459 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-78f4jgdp7"] Oct 30 22:19:50.213709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:50.213668 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7aac589-0369-4952-be8b-5ee693d6db8b" path="/var/lib/kubelet/pods/b7aac589-0369-4952-be8b-5ee693d6db8b/volumes" Oct 30 22:19:58.947906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:58.947868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:19:59.983384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.983347 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:19:59.984408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.984385 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b7aac589-0369-4952-be8b-5ee693d6db8b" containerName="payload-processing" Oct 30 22:19:59.984408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.984408 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7aac589-0369-4952-be8b-5ee693d6db8b" containerName="payload-processing" Oct 30 22:19:59.984686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.984669 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7aac589-0369-4952-be8b-5ee693d6db8b" containerName="payload-processing" Oct 30 22:19:59.990454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.990425 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:19:59.995254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995229 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-1fd54e0920e6\"" Oct 30 22:19:59.995476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995367 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\"" Oct 30 22:19:59.995476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995399 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\"" Oct 30 22:19:59.995602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995481 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\"" Oct 30 22:19:59.995602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995526 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\"" Oct 30 22:19:59.995753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:19:59.995667 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-fkzjt\"" Oct 30 22:20:00.002556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.002530 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:20:00.134319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.128287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlzp\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.134319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.128385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.134319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.128419 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.134319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.128572 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.242221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.233525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.242221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.233829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlzp\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.242221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.233902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.242221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.233933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.242221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.234774 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.257287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.253642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.269397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.268498 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.303293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.302241 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlzp\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.326233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.324782 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:00.649291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.636001 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:20:00.718572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.718531 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.726856 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"openshift-service-ca.crt\"" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.726964 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1a5bdf\"" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.727016 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"routing-overlay\"" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.727116 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"agc-cred-2b0fc222dc2a\"" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.727364 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"payload-processing-external-model-e2e--378e44ca-dockercfg-z8ckg\"" Oct 30 22:20:00.730888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.727486 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"kube-root-ca.crt\"" Oct 30 22:20:00.745126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.735507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.745663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.745636 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.749467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.746597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmw4\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.749722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.749689 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.754094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.754066 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:20:00.960305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.960091 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.960305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.960231 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmw4\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.960305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.960309 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.960701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.960484 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:00.994291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:00.986077 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:01.042343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.019296 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:01.100016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.099984 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:01.119392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.119330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmw4\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:01.136705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.136674 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm"] Oct 30 22:20:01.144836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.144804 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b02ad45-mhpgm"] Oct 30 22:20:01.185779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.175142 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:20:01.348234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.348207 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:01.659502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:01.659468 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:20:01.660245 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:20:01.660220 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808605bf_d216_480f_881d_641fd650b49d.slice/crio-dff962ac229a7db0cdc406aff35f03dc081bdf2ceae16f07991fcec36ba96346 WatchSource:0}: Error finding container dff962ac229a7db0cdc406aff35f03dc081bdf2ceae16f07991fcec36ba96346: Status 404 returned error can't find the container with id dff962ac229a7db0cdc406aff35f03dc081bdf2ceae16f07991fcec36ba96346 Oct 30 22:20:02.191057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.191023 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd1153d-da27-45b4-8da6-2f9a20f68348" path="/var/lib/kubelet/pods/5cd1153d-da27-45b4-8da6-2f9a20f68348/volumes" Oct 30 22:20:02.251491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.251455 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" event={"ID":"808605bf-d216-480f-881d-641fd650b49d","Type":"ContainerStarted","Data":"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07"} Oct 30 22:20:02.251684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.251517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" event={"ID":"808605bf-d216-480f-881d-641fd650b49d","Type":"ContainerStarted","Data":"dff962ac229a7db0cdc406aff35f03dc081bdf2ceae16f07991fcec36ba96346"} Oct 30 22:20:02.251684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.251546 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:02.253301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.253238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" event={"ID":"024ab905-32c6-4093-9149-989d38fb290c","Type":"ContainerStarted","Data":"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91"} Oct 30 22:20:02.253301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.253298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" event={"ID":"024ab905-32c6-4093-9149-989d38fb290c","Type":"ContainerStarted","Data":"4e408fa2442127492815cf26f460b059fcfcdf1c027c41db0b4a2f03f29d528b"} Oct 30 22:20:02.253475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.253320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:02.294048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.293996 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" podStartSLOduration=2.293983257 podStartE2EDuration="2.293983257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.289737249 +0000 UTC m=+2426.239692195" watchObservedRunningTime="2026-10-30 22:20:02.293983257 +0000 UTC m=+2426.243938194" Oct 30 22:20:02.338609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:02.338554 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" podStartSLOduration=3.338537462 podStartE2EDuration="3.338537462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.335037586 +0000 UTC m=+2426.284992539" watchObservedRunningTime="2026-10-30 22:20:02.338537462 +0000 UTC m=+2426.288492443" Oct 30 22:20:10.217533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:10.217492 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 22:20:10.218220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:10.217737 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" containerID="cri-o://a43541d9c78265932433ff9c48bb8d5ddeb10b89d088603f67a256888f88ccd1" gracePeriod=30 Oct 30 22:20:12.034132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:12.034096 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx"] Oct 30 22:20:12.039648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:12.039612 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831cd94-q69hx"] Oct 30 22:20:12.207819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:12.207781 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ab6290-d5d2-4a45-84cf-13ae2a85c6a0" path="/var/lib/kubelet/pods/76ab6290-d5d2-4a45-84cf-13ae2a85c6a0/volumes" Oct 30 22:20:13.269006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:13.268975 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:20:13.269461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:13.269059 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:20:15.518480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.518444 5036 generic.go:356] "Generic (PLEG): container finished" podID="373af11a-0efb-4536-abba-e9246f5aedb2" containerID="a43541d9c78265932433ff9c48bb8d5ddeb10b89d088603f67a256888f88ccd1" exitCode=0 Oct 30 22:20:15.518858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.518520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" event={"ID":"373af11a-0efb-4536-abba-e9246f5aedb2","Type":"ContainerDied","Data":"a43541d9c78265932433ff9c48bb8d5ddeb10b89d088603f67a256888f88ccd1"} Oct 30 22:20:15.518858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.518570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" event={"ID":"373af11a-0efb-4536-abba-e9246f5aedb2","Type":"ContainerDied","Data":"52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd"} Oct 30 22:20:15.518858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.518585 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ee879888a9b10b832922817d87bb32f88ae1be6acef18914a3d57485858bfd" Oct 30 22:20:15.538374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.538348 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 22:20:15.688550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688513 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.688706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.688768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.688768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688753 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.688839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688782 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.688839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.688945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.688929 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.689055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689038 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.689154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689091 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.689154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689132 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\") pod \"373af11a-0efb-4536-abba-e9246f5aedb2\" (UID: \"373af11a-0efb-4536-abba-e9246f5aedb2\") " Oct 30 22:20:15.689154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.689356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689155 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.689356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:15.689589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689489 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.689961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689939 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373af11a-0efb-4536-abba-e9246f5aedb2-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.690058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689967 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.690058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689982 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.690058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.689998 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.690058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.690013 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.691749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.691724 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.691840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.691730 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:15.691992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.691966 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:15.692086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.692054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz" pod "373af11a-0efb-4536-abba-e9246f5aedb2" (UID: "373af11a-0efb-4536-abba-e9246f5aedb2"). InnerVolumeSpecName "kube-api-access-5pcqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:15.797376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.795834 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373af11a-0efb-4536-abba-e9246f5aedb2-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.797376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.795876 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcqz\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-kube-api-access-5pcqz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.797376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.795893 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373af11a-0efb-4536-abba-e9246f5aedb2-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.797376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:15.795905 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373af11a-0efb-4536-abba-e9246f5aedb2-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.543305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:16.543196 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7" Oct 30 22:20:16.584742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:16.584695 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 22:20:16.590913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:16.590877 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-zrtn7"] Oct 30 22:20:18.103627 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:20:18.103586 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:20:18.192662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:18.192628 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" path="/var/lib/kubelet/pods/373af11a-0efb-4536-abba-e9246f5aedb2/volumes" Oct 30 22:20:48.101236 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:20:48.101161 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:20:48.295346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:48.295314 5036 scope.go:122] "RemoveContainer" containerID="1476dbbc48e8eda240f3114516643de2913822ab4d8a412bde3e25c6f9768dd0" Oct 30 22:20:48.307313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:48.307291 5036 scope.go:122] "RemoveContainer" containerID="30d57e16e8d6333fa8692eb3a6fe5430eb506e0fc5455a18abba11711a38645e" Oct 30 22:20:48.318353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:48.318329 5036 scope.go:122] "RemoveContainer" containerID="51c3844ceae43b11acaf1c48c18887e6324631bdd6a18c03c1ec0e7a2e2373f2" Oct 30 22:20:48.387729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:48.387678 5036 scope.go:122] "RemoveContainer" containerID="a43541d9c78265932433ff9c48bb8d5ddeb10b89d088603f67a256888f88ccd1" Oct 30 22:20:48.412054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:48.412006 5036 scope.go:122] "RemoveContainer" containerID="e9f8edb6072dcc71865065416899f36252be99f24b1677bded547a294212de18" Oct 30 22:20:53.248339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.248307 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd"] Oct 30 22:20:53.249219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.249201 5036 cpu_manager.go:414] "containerMap: removing container" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" Oct 30 22:20:53.249284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.249221 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" Oct 30 22:20:53.249401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.249391 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="373af11a-0efb-4536-abba-e9246f5aedb2" containerName="istio-proxy" Oct 30 22:20:53.255140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.255121 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.259954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.259930 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:53.260098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.260026 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:53.260098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.260063 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:53.261051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.261027 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:20:53.263100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.263077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd"] Oct 30 22:20:53.370116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.370068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.370781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.370752 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbj5r\" (UniqueName: \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.375817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.375791 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.477622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.477589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.477811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.477726 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.477886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.477874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbj5r\" (UniqueName: \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.478435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.478363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.480148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.480125 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.490500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.490463 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbj5r\" (UniqueName: \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\") pod \"maas-api-revoke-sub-504c1f76-9k9qd\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.575343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.575253 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:53.866431 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:20:53.866399 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89e1157_560b_4c02_9396_0ac8b9ecc02a.slice/crio-0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46 WatchSource:0}: Error finding container 0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46: Status 404 returned error can't find the container with id 0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46 Oct 30 22:20:53.866572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:53.866492 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd"] Oct 30 22:20:54.109499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:54.109459 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" event={"ID":"f89e1157-560b-4c02-9396-0ac8b9ecc02a","Type":"ContainerStarted","Data":"0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46"} Oct 30 22:20:55.122565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:55.122531 5036 generic.go:356] "Generic (PLEG): container finished" podID="f89e1157-560b-4c02-9396-0ac8b9ecc02a" containerID="e9b257a95e7ddd3e671c9c8924e3d0a6eba7d9bf8c13fb8fede3ebb91579ec93" exitCode=0 Oct 30 22:20:55.122991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:55.122615 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" event={"ID":"f89e1157-560b-4c02-9396-0ac8b9ecc02a","Type":"ContainerDied","Data":"e9b257a95e7ddd3e671c9c8924e3d0a6eba7d9bf8c13fb8fede3ebb91579ec93"} Oct 30 22:20:56.330943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.330905 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:56.432802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.432712 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\") pod \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " Oct 30 22:20:56.433100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.433079 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\" (UniqueName: \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\") pod \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " Oct 30 22:20:56.433223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.433089 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca" pod "f89e1157-560b-4c02-9396-0ac8b9ecc02a" (UID: "f89e1157-560b-4c02-9396-0ac8b9ecc02a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:56.433223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.433149 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\") pod \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\" (UID: \"f89e1157-560b-4c02-9396-0ac8b9ecc02a\") " Oct 30 22:20:56.434015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.433990 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:56.435481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.435458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r" pod "f89e1157-560b-4c02-9396-0ac8b9ecc02a" (UID: "f89e1157-560b-4c02-9396-0ac8b9ecc02a"). InnerVolumeSpecName "kube-api-access-hbj5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:56.435612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.435593 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca" pod "f89e1157-560b-4c02-9396-0ac8b9ecc02a" (UID: "f89e1157-560b-4c02-9396-0ac8b9ecc02a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:56.535382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.535339 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbj5r\" (UniqueName: \"kubernetes.io/projected/f89e1157-560b-4c02-9396-0ac8b9ecc02a-kube-api-access-hbj5r\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:56.535382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:56.535377 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89e1157-560b-4c02-9396-0ac8b9ecc02a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:57.144859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:57.144824 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" Oct 30 22:20:57.145045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:57.144823 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd" event={"ID":"f89e1157-560b-4c02-9396-0ac8b9ecc02a","Type":"ContainerDied","Data":"0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46"} Oct 30 22:20:57.145045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:57.144958 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a4b77c33fe16038a93341acef9c151c80ff0c2b51280f667d5072531e734d46" Oct 30 22:20:59.029508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:59.029474 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr"] Oct 30 22:20:59.034777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:20:59.034746 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c92d3d-mvshr"] Oct 30 22:21:00.207409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:00.207372 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5839a362-8356-4ce1-a2b9-ade8c5b39d82" path="/var/lib/kubelet/pods/5839a362-8356-4ce1-a2b9-ade8c5b39d82/volumes" Oct 30 22:21:04.395284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:04.395231 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:21:05.006136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.006099 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl"] Oct 30 22:21:05.007075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.007055 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f89e1157-560b-4c02-9396-0ac8b9ecc02a" containerName="revoke-keys" Oct 30 22:21:05.007155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.007077 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89e1157-560b-4c02-9396-0ac8b9ecc02a" containerName="revoke-keys" Oct 30 22:21:05.007213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.007204 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89e1157-560b-4c02-9396-0ac8b9ecc02a" containerName="revoke-keys" Oct 30 22:21:05.011425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.011403 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.018621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.018595 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:21:05.018893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.018870 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:21:05.019237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.019212 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:21:05.020986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.020963 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:21:05.021113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.021048 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl"] Oct 30 22:21:05.122788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.122755 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cf6s\" (UniqueName: \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.123023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.122847 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.123095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.123074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.239561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.227650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cf6s\" (UniqueName: \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.239561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.227790 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.239561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.228042 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.244337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.243155 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.252014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.245131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.273820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.271343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cf6s\" (UniqueName: \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\") pod \"maas-api-revoke-sub-b601d01b-22dhl\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.334430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.334401 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:05.639353 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:21:05.639321 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4865a7_34c4_4a01_bebd_cb33f29e9a16.slice/crio-64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9 WatchSource:0}: Error finding container 64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9: Status 404 returned error can't find the container with id 64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9 Oct 30 22:21:05.639868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:05.639845 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl"] Oct 30 22:21:06.305864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:06.305777 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" event={"ID":"2c4865a7-34c4-4a01-bebd-cb33f29e9a16","Type":"ContainerStarted","Data":"62f26994718f5b7d3bfecb3550e9bad1d894828203ec48971f35a71568f612ca"} Oct 30 22:21:06.305864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:06.305848 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" event={"ID":"2c4865a7-34c4-4a01-bebd-cb33f29e9a16","Type":"ContainerStarted","Data":"64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9"} Oct 30 22:21:06.330564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:06.330499 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" podStartSLOduration=2.330479768 podStartE2EDuration="2.330479768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:06.322621824 +0000 UTC m=+2490.272576771" watchObservedRunningTime="2026-10-30 22:21:06.330479768 +0000 UTC m=+2490.280434714" Oct 30 22:21:07.327289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:07.326994 5036 generic.go:356] "Generic (PLEG): container finished" podID="2c4865a7-34c4-4a01-bebd-cb33f29e9a16" containerID="62f26994718f5b7d3bfecb3550e9bad1d894828203ec48971f35a71568f612ca" exitCode=0 Oct 30 22:21:07.327289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:07.327066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" event={"ID":"2c4865a7-34c4-4a01-bebd-cb33f29e9a16","Type":"ContainerDied","Data":"62f26994718f5b7d3bfecb3550e9bad1d894828203ec48971f35a71568f612ca"} Oct 30 22:21:08.542042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.542011 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:08.647349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.642049 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\") pod \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " Oct 30 22:21:08.647349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.642259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\" (UniqueName: \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\") pod \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " Oct 30 22:21:08.647349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.642382 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\") pod \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\" (UID: \"2c4865a7-34c4-4a01-bebd-cb33f29e9a16\") " Oct 30 22:21:08.647349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.645977 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca" pod "2c4865a7-34c4-4a01-bebd-cb33f29e9a16" (UID: "2c4865a7-34c4-4a01-bebd-cb33f29e9a16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:08.669519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.669480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s" pod "2c4865a7-34c4-4a01-bebd-cb33f29e9a16" (UID: "2c4865a7-34c4-4a01-bebd-cb33f29e9a16"). InnerVolumeSpecName "kube-api-access-5cf6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:08.671740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.669609 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca" pod "2c4865a7-34c4-4a01-bebd-cb33f29e9a16" (UID: "2c4865a7-34c4-4a01-bebd-cb33f29e9a16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:08.746785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.746755 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cf6s\" (UniqueName: \"kubernetes.io/projected/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-kube-api-access-5cf6s\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:08.746785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.746785 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:08.746987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:08.746795 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4865a7-34c4-4a01-bebd-cb33f29e9a16-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:09.389346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.387501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" event={"ID":"2c4865a7-34c4-4a01-bebd-cb33f29e9a16","Type":"ContainerDied","Data":"64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9"} Oct 30 22:21:09.389346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.387574 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64288f7e3e2db4d63fed36e934af89d1f58c7f0a9eebebd55af9993a5106f2b9" Oct 30 22:21:09.389346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.387642 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl" Oct 30 22:21:09.395679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.395638 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" podUID="90fc6654-747d-43cf-a178-0919a87344b2" containerName="payload-processing" containerID="cri-o://9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7" gracePeriod=25 Oct 30 22:21:09.722729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.722701 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:21:09.782499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.782459 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\") pod \"90fc6654-747d-43cf-a178-0919a87344b2\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " Oct 30 22:21:09.782681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.782663 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\") pod \"90fc6654-747d-43cf-a178-0919a87344b2\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " Oct 30 22:21:09.782825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.782805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\") pod \"90fc6654-747d-43cf-a178-0919a87344b2\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " Oct 30 22:21:09.782950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.782931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\") pod \"90fc6654-747d-43cf-a178-0919a87344b2\" (UID: \"90fc6654-747d-43cf-a178-0919a87344b2\") " Oct 30 22:21:09.783181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.783092 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume" pod "90fc6654-747d-43cf-a178-0919a87344b2" (UID: "90fc6654-747d-43cf-a178-0919a87344b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:09.783640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.783610 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume" pod "90fc6654-747d-43cf-a178-0919a87344b2" (UID: "90fc6654-747d-43cf-a178-0919a87344b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:09.784735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.784704 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:09.784937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.784739 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90fc6654-747d-43cf-a178-0919a87344b2-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:09.785598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.785569 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials" pod "90fc6654-747d-43cf-a178-0919a87344b2" (UID: "90fc6654-747d-43cf-a178-0919a87344b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:09.786371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.786343 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh" pod "90fc6654-747d-43cf-a178-0919a87344b2" (UID: "90fc6654-747d-43cf-a178-0919a87344b2"). InnerVolumeSpecName "kube-api-access-fvjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:09.887140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.887105 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:09.887140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:09.887141 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjdh\" (UniqueName: \"kubernetes.io/projected/90fc6654-747d-43cf-a178-0919a87344b2-kube-api-access-fvjdh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.400517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.400479 5036 generic.go:356] "Generic (PLEG): container finished" podID="90fc6654-747d-43cf-a178-0919a87344b2" containerID="9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7" exitCode=0 Oct 30 22:21:10.400517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.400518 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" Oct 30 22:21:10.400805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.400565 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" event={"ID":"90fc6654-747d-43cf-a178-0919a87344b2","Type":"ContainerDied","Data":"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7"} Oct 30 22:21:10.400805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.400628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9" event={"ID":"90fc6654-747d-43cf-a178-0919a87344b2","Type":"ContainerDied","Data":"fcb59d8474a78c37986351494a0a3993b327684a3cfb8b762ece354a191b0f20"} Oct 30 22:21:10.400805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.400660 5036 scope.go:122] "RemoveContainer" containerID="9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7" Oct 30 22:21:10.413908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.413892 5036 scope.go:122] "RemoveContainer" containerID="9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7" Oct 30 22:21:10.414178 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:21:10.414160 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7\": container with ID starting with 9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7 not found: ID does not exist" containerID="9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7" Oct 30 22:21:10.414250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.414186 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7"} err="failed to get container status \"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7\": rpc error: code = NotFound desc = could not find container \"9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7\": container with ID starting with 9dbf6936e4d96abaa916dc1972094c7d3475132818a9946005c965ee7b45c5e7 not found: ID does not exist" Oct 30 22:21:10.422253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.422223 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:21:10.428189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:10.428164 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-558cqfcd9"] Oct 30 22:21:12.193180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:12.193145 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fc6654-747d-43cf-a178-0919a87344b2" path="/var/lib/kubelet/pods/90fc6654-747d-43cf-a178-0919a87344b2/volumes" Oct 30 22:21:16.100135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:16.100100 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:21:18.100866 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:21:18.100826 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:21:21.101088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.101045 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" podUID="13718ae8-7811-45d7-9698-1978db6e8498" containerName="payload-processing" containerID="cri-o://79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813" gracePeriod=25 Oct 30 22:21:21.437057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.437025 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:21:21.612846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.612816 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\") pod \"13718ae8-7811-45d7-9698-1978db6e8498\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " Oct 30 22:21:21.613067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.612972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\") pod \"13718ae8-7811-45d7-9698-1978db6e8498\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " Oct 30 22:21:21.613169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.613154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\") pod \"13718ae8-7811-45d7-9698-1978db6e8498\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " Oct 30 22:21:21.613278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.613248 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\") pod \"13718ae8-7811-45d7-9698-1978db6e8498\" (UID: \"13718ae8-7811-45d7-9698-1978db6e8498\") " Oct 30 22:21:21.613349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.613322 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume" pod "13718ae8-7811-45d7-9698-1978db6e8498" (UID: "13718ae8-7811-45d7-9698-1978db6e8498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:21.613863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.613836 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume" pod "13718ae8-7811-45d7-9698-1978db6e8498" (UID: "13718ae8-7811-45d7-9698-1978db6e8498"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:21.615978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.614208 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:21.616151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.616080 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials" pod "13718ae8-7811-45d7-9698-1978db6e8498" (UID: "13718ae8-7811-45d7-9698-1978db6e8498"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:21.618803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.618778 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5" pod "13718ae8-7811-45d7-9698-1978db6e8498" (UID: "13718ae8-7811-45d7-9698-1978db6e8498"). InnerVolumeSpecName "kube-api-access-cq2b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:21.695350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.695213 5036 generic.go:356] "Generic (PLEG): container finished" podID="13718ae8-7811-45d7-9698-1978db6e8498" containerID="79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813" exitCode=0 Oct 30 22:21:21.695350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.695258 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" Oct 30 22:21:21.695350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.695300 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" event={"ID":"13718ae8-7811-45d7-9698-1978db6e8498","Type":"ContainerDied","Data":"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813"} Oct 30 22:21:21.695668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.695357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8" event={"ID":"13718ae8-7811-45d7-9698-1978db6e8498","Type":"ContainerDied","Data":"03dc43baae7c61dec1042d05891249ac467f17957741fd383188229374cd35db"} Oct 30 22:21:21.695668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.695374 5036 scope.go:122] "RemoveContainer" containerID="79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813" Oct 30 22:21:21.710597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.710573 5036 scope.go:122] "RemoveContainer" containerID="79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813" Oct 30 22:21:21.711077 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:21:21.710988 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813\": container with ID starting with 79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813 not found: ID does not exist" containerID="79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813" Oct 30 22:21:21.711077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.711030 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813"} err="failed to get container status \"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813\": rpc error: code = NotFound desc = could not find container \"79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813\": container with ID starting with 79eb7b2d14cfbb74ed26df6aef6ae3b35605dd3164e28259cf0d2d392a8c4813 not found: ID does not exist" Oct 30 22:21:21.717208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.717178 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13718ae8-7811-45d7-9698-1978db6e8498-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:21.717385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.717210 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2b5\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-kube-api-access-cq2b5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:21.717385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.717226 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13718ae8-7811-45d7-9698-1978db6e8498-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:21.728970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.728939 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:21:21.736192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:21.736163 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-65b8hjmn8"] Oct 30 22:21:22.197481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:22.197434 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13718ae8-7811-45d7-9698-1978db6e8498" path="/var/lib/kubelet/pods/13718ae8-7811-45d7-9698-1978db6e8498/volumes" Oct 30 22:21:26.503142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.503112 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:26.503939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.503922 5036 cpu_manager.go:414] "containerMap: removing container" podUID="90fc6654-747d-43cf-a178-0919a87344b2" containerName="payload-processing" Oct 30 22:21:26.503992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.503940 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fc6654-747d-43cf-a178-0919a87344b2" containerName="payload-processing" Oct 30 22:21:26.503992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.503959 5036 cpu_manager.go:414] "containerMap: removing container" podUID="13718ae8-7811-45d7-9698-1978db6e8498" containerName="payload-processing" Oct 30 22:21:26.503992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.503964 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13718ae8-7811-45d7-9698-1978db6e8498" containerName="payload-processing" Oct 30 22:21:26.504093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.504018 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2c4865a7-34c4-4a01-bebd-cb33f29e9a16" containerName="revoke-keys" Oct 30 22:21:26.504093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.504023 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4865a7-34c4-4a01-bebd-cb33f29e9a16" containerName="revoke-keys" Oct 30 22:21:26.504175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.504133 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4865a7-34c4-4a01-bebd-cb33f29e9a16" containerName="revoke-keys" Oct 30 22:21:26.504175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.504143 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fc6654-747d-43cf-a178-0919a87344b2" containerName="payload-processing" Oct 30 22:21:26.504175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.504152 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="13718ae8-7811-45d7-9698-1978db6e8498" containerName="payload-processing" Oct 30 22:21:26.509613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.509592 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.522594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.522569 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-e06d3afbed37\"" Oct 30 22:21:26.522745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.522638 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"openshift-service-ca.crt\"" Oct 30 22:21:26.522843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.522816 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"kube-root-ca.crt\"" Oct 30 22:21:26.523013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.522890 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"routing-overlay\"" Oct 30 22:21:26.524065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.524042 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9d868\"" Oct 30 22:21:26.529023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.529002 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:26.569686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.569650 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.569939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.569917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.570583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.570555 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.570692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.570654 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjwr\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.676289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.675935 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.676289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.676181 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.676289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.676258 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.676652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.676373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjwr\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.684302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.678200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.698530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.698497 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.701802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.701771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjwr\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.712294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.707064 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:26.829672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:26.829580 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:27.240798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.240763 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:27.263568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.263533 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.263965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.263940 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:27.270918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.270892 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a5f192\"" Oct 30 22:21:27.271102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.271014 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"routing-overlay\"" Oct 30 22:21:27.271164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.271119 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-03ae22d0bf6c\"" Oct 30 22:21:27.271758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.271734 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"kube-root-ca.crt\"" Oct 30 22:21:27.271932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.271918 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"openshift-service-ca.crt\"" Oct 30 22:21:27.271994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.271952 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"payload-processing-external-model-e2e--fed5953e-dockercfg-fzhht\"" Oct 30 22:21:27.292029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.291934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhl98\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.292029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.292031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.292929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.292191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.342332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.337667 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.497179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.497084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.497646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.497620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.499217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.499089 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.499510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.499487 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhl98\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.499666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.499648 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.507044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.500560 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.507541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.505142 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.540384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.540010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhl98\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.592979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.592949 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:27.798009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.797978 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:27.798402 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:21:27.798380 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322a84c3_80e0_46bc_b6d2_b135f3171855.slice/crio-c6663201f7b1181ff1544eb4683d1d2478fed7f44b2746dcf117a67091c00f43 WatchSource:0}: Error finding container c6663201f7b1181ff1544eb4683d1d2478fed7f44b2746dcf117a67091c00f43: Status 404 returned error can't find the container with id c6663201f7b1181ff1544eb4683d1d2478fed7f44b2746dcf117a67091c00f43 Oct 30 22:21:27.834027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.833992 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" event={"ID":"322a84c3-80e0-46bc-b6d2-b135f3171855","Type":"ContainerStarted","Data":"c6663201f7b1181ff1544eb4683d1d2478fed7f44b2746dcf117a67091c00f43"} Oct 30 22:21:27.907428 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:21:27.907396 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e419b3_b295_47c1_bd79_5418afee5d9c.slice/crio-987c66ab2a9cf4f60eebab1973d59a1fa9dfba1a65d674d477543b9aa33d3e32 WatchSource:0}: Error finding container 987c66ab2a9cf4f60eebab1973d59a1fa9dfba1a65d674d477543b9aa33d3e32: Status 404 returned error can't find the container with id 987c66ab2a9cf4f60eebab1973d59a1fa9dfba1a65d674d477543b9aa33d3e32 Oct 30 22:21:27.908357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:27.908332 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:28.848503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.848463 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" event={"ID":"322a84c3-80e0-46bc-b6d2-b135f3171855","Type":"ContainerStarted","Data":"d5c12cead5edf32927a6cc1a4a5478815266b0475842bf7b16e6b15135b8f483"} Oct 30 22:21:28.849028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.848537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:28.850623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.850594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" event={"ID":"c2e419b3-b295-47c1-bd79-5418afee5d9c","Type":"ContainerStarted","Data":"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504"} Oct 30 22:21:28.850777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.850644 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" event={"ID":"c2e419b3-b295-47c1-bd79-5418afee5d9c","Type":"ContainerStarted","Data":"987c66ab2a9cf4f60eebab1973d59a1fa9dfba1a65d674d477543b9aa33d3e32"} Oct 30 22:21:28.850777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.850678 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:28.871128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.871074 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" podStartSLOduration=2.8710582540000003 podStartE2EDuration="2.871058254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:28.8671844 +0000 UTC m=+2512.817139346" watchObservedRunningTime="2026-10-30 22:21:28.871058254 +0000 UTC m=+2512.821013201" Oct 30 22:21:28.894729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:28.894674 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" podStartSLOduration=1.894659502 podStartE2EDuration="1.894659502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:28.893094911 +0000 UTC m=+2512.843049857" watchObservedRunningTime="2026-10-30 22:21:28.894659502 +0000 UTC m=+2512.844614448" Oct 30 22:21:32.433799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.433747 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:21:32.440395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.440352 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.442949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.442693 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:21:32.443884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.443860 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-ada4a473a1cc\"" Oct 30 22:21:32.546577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.546540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.546793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.546687 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gcv\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.546793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.546773 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.546936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.546921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.648428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.648396 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.648609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.648518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gcv\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.648609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.648540 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.648706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.648643 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.649140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.649113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.649387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.649366 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.651427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.651405 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.660786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.660759 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gcv\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.733127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.733041 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:21:32.738466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.738442 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.741799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.741774 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-3808c12e12a9\"" Oct 30 22:21:32.741982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.741961 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"payload-processing-external-model-e2e--2051f056-dockercfg-9hgtp\"" Oct 30 22:21:32.749340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.749315 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:21:32.764519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.764496 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:32.856549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.851664 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.856549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.851813 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.856549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.851920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.856549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.851946 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4cb\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.953665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.953623 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.953873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.953753 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.953873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.953867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.954009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.953904 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4cb\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.954515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.954475 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.954783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.954755 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.956969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.956937 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:32.966612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:32.966583 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4cb\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:33.060841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.060764 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:33.073084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.073059 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:21:33.076904 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:21:33.076877 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6fa369_5cc9_409b_b6fb_eb3bf82ae609.slice/crio-f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513 WatchSource:0}: Error finding container f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513: Status 404 returned error can't find the container with id f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513 Oct 30 22:21:33.423247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.423214 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:21:33.423530 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:21:33.423506 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1876bc34_eeac_4a72_9c72_b8ac23e89d82.slice/crio-3caf22c2ae911d206d9d1a51f30efeb68f972965407a5965f497257c60742f7f WatchSource:0}: Error finding container 3caf22c2ae911d206d9d1a51f30efeb68f972965407a5965f497257c60742f7f: Status 404 returned error can't find the container with id 3caf22c2ae911d206d9d1a51f30efeb68f972965407a5965f497257c60742f7f Oct 30 22:21:33.981219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.981178 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" event={"ID":"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609","Type":"ContainerStarted","Data":"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37"} Oct 30 22:21:33.981871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.981249 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" event={"ID":"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609","Type":"ContainerStarted","Data":"f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513"} Oct 30 22:21:33.982162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.982139 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:33.987317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.984840 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" event={"ID":"1876bc34-eeac-4a72-9c72-b8ac23e89d82","Type":"ContainerStarted","Data":"481940e0d1ca697e9fc0b7c161f176ff63f3fd8dcba5b7debd4bc078ed983a17"} Oct 30 22:21:33.987317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.984890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" event={"ID":"1876bc34-eeac-4a72-9c72-b8ac23e89d82","Type":"ContainerStarted","Data":"3caf22c2ae911d206d9d1a51f30efeb68f972965407a5965f497257c60742f7f"} Oct 30 22:21:33.987317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:33.984998 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:34.003700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:34.003652 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" podStartSLOduration=2.003636035 podStartE2EDuration="2.003636035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:34.001878347 +0000 UTC m=+2517.951833294" watchObservedRunningTime="2026-10-30 22:21:34.003636035 +0000 UTC m=+2517.953590981" Oct 30 22:21:34.019069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:34.019027 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" podStartSLOduration=2.019014812 podStartE2EDuration="2.019014812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:34.018835434 +0000 UTC m=+2517.968790381" watchObservedRunningTime="2026-10-30 22:21:34.019014812 +0000 UTC m=+2517.968969759" Oct 30 22:21:39.865328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:39.865258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:39.865730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:39.865371 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:44.998453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:44.998422 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:21:45.036312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:45.035949 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn"] Oct 30 22:21:45.044041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:45.044008 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d818aa0c-gt8bn"] Oct 30 22:21:45.079312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:45.079279 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:46.043504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:46.043468 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:21:46.300346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:46.299757 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb3042b-235e-4733-ab8e-4552b525905d" path="/var/lib/kubelet/pods/9fb3042b-235e-4733-ab8e-4552b525905d/volumes" Oct 30 22:21:46.300535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:46.300367 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:48.101628 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:21:48.101582 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:21:48.489282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:48.489229 5036 scope.go:122] "RemoveContainer" containerID="b53622715a967bdb77e6b8df3ff35ab91f0145ca6f27493bd52e648d1a9ce663" Oct 30 22:21:48.501073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:48.501052 5036 scope.go:122] "RemoveContainer" containerID="a8c56f641e34972df11bf31e5219c000d40d72088c85c62986e5d91b3e184388" Oct 30 22:21:50.092289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.089770 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" podUID="322a84c3-80e0-46bc-b6d2-b135f3171855" containerName="payload-processing" containerID="cri-o://d5c12cead5edf32927a6cc1a4a5478815266b0475842bf7b16e6b15135b8f483" gracePeriod=25 Oct 30 22:21:50.245432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.245396 5036 generic.go:356] "Generic (PLEG): container finished" podID="322a84c3-80e0-46bc-b6d2-b135f3171855" containerID="d5c12cead5edf32927a6cc1a4a5478815266b0475842bf7b16e6b15135b8f483" exitCode=0 Oct 30 22:21:50.245622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.245439 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" event={"ID":"322a84c3-80e0-46bc-b6d2-b135f3171855","Type":"ContainerDied","Data":"d5c12cead5edf32927a6cc1a4a5478815266b0475842bf7b16e6b15135b8f483"} Oct 30 22:21:50.421650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.421624 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:50.544608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.544567 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\") pod \"322a84c3-80e0-46bc-b6d2-b135f3171855\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " Oct 30 22:21:50.544784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.544684 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\") pod \"322a84c3-80e0-46bc-b6d2-b135f3171855\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " Oct 30 22:21:50.544838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.544821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\") pod \"322a84c3-80e0-46bc-b6d2-b135f3171855\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " Oct 30 22:21:50.545001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.544982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\") pod \"322a84c3-80e0-46bc-b6d2-b135f3171855\" (UID: \"322a84c3-80e0-46bc-b6d2-b135f3171855\") " Oct 30 22:21:50.545084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.545059 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume" pod "322a84c3-80e0-46bc-b6d2-b135f3171855" (UID: "322a84c3-80e0-46bc-b6d2-b135f3171855"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:50.545419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.545388 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume" pod "322a84c3-80e0-46bc-b6d2-b135f3171855" (UID: "322a84c3-80e0-46bc-b6d2-b135f3171855"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:50.545838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.545817 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:50.545921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.545842 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/322a84c3-80e0-46bc-b6d2-b135f3171855-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:50.547562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.547540 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials" pod "322a84c3-80e0-46bc-b6d2-b135f3171855" (UID: "322a84c3-80e0-46bc-b6d2-b135f3171855"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:50.547646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.547620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr" pod "322a84c3-80e0-46bc-b6d2-b135f3171855" (UID: "322a84c3-80e0-46bc-b6d2-b135f3171855"). InnerVolumeSpecName "kube-api-access-bpjwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:50.647741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.647703 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:50.647741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:50.647736 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjwr\" (UniqueName: \"kubernetes.io/projected/322a84c3-80e0-46bc-b6d2-b135f3171855-kube-api-access-bpjwr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:51.259498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.259464 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" Oct 30 22:21:51.260020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.259470 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk" event={"ID":"322a84c3-80e0-46bc-b6d2-b135f3171855","Type":"ContainerDied","Data":"c6663201f7b1181ff1544eb4683d1d2478fed7f44b2746dcf117a67091c00f43"} Oct 30 22:21:51.260020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.259622 5036 scope.go:122] "RemoveContainer" containerID="d5c12cead5edf32927a6cc1a4a5478815266b0475842bf7b16e6b15135b8f483" Oct 30 22:21:51.288854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.288821 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:51.294170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.294147 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649ch8rqk"] Oct 30 22:21:51.300882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.300852 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" podUID="c2e419b3-b295-47c1-bd79-5418afee5d9c" containerName="payload-processing" containerID="cri-o://8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504" gracePeriod=25 Oct 30 22:21:51.644375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.644343 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:51.766864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.766780 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\") pod \"c2e419b3-b295-47c1-bd79-5418afee5d9c\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " Oct 30 22:21:51.767066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.766908 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\") pod \"c2e419b3-b295-47c1-bd79-5418afee5d9c\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " Oct 30 22:21:51.767066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.766976 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\") pod \"c2e419b3-b295-47c1-bd79-5418afee5d9c\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " Oct 30 22:21:51.767066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.767029 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\") pod \"c2e419b3-b295-47c1-bd79-5418afee5d9c\" (UID: \"c2e419b3-b295-47c1-bd79-5418afee5d9c\") " Oct 30 22:21:51.767362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.767332 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume" pod "c2e419b3-b295-47c1-bd79-5418afee5d9c" (UID: "c2e419b3-b295-47c1-bd79-5418afee5d9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:51.767514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.767435 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume" pod "c2e419b3-b295-47c1-bd79-5418afee5d9c" (UID: "c2e419b3-b295-47c1-bd79-5418afee5d9c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:51.767873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.767851 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:51.767958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.767875 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e419b3-b295-47c1-bd79-5418afee5d9c-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:51.769309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.769257 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98" pod "c2e419b3-b295-47c1-bd79-5418afee5d9c" (UID: "c2e419b3-b295-47c1-bd79-5418afee5d9c"). InnerVolumeSpecName "kube-api-access-zhl98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:51.769635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.769612 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials" pod "c2e419b3-b295-47c1-bd79-5418afee5d9c" (UID: "c2e419b3-b295-47c1-bd79-5418afee5d9c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:51.869398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.869356 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhl98\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-kube-api-access-zhl98\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:51.869398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:51.869398 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e419b3-b295-47c1-bd79-5418afee5d9c-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:52.190564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.190532 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322a84c3-80e0-46bc-b6d2-b135f3171855" path="/var/lib/kubelet/pods/322a84c3-80e0-46bc-b6d2-b135f3171855/volumes" Oct 30 22:21:52.274797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.274764 5036 generic.go:356] "Generic (PLEG): container finished" podID="c2e419b3-b295-47c1-bd79-5418afee5d9c" containerID="8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504" exitCode=0 Oct 30 22:21:52.275310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.274806 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" Oct 30 22:21:52.275310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.274845 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" event={"ID":"c2e419b3-b295-47c1-bd79-5418afee5d9c","Type":"ContainerDied","Data":"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504"} Oct 30 22:21:52.275310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.274909 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr" event={"ID":"c2e419b3-b295-47c1-bd79-5418afee5d9c","Type":"ContainerDied","Data":"987c66ab2a9cf4f60eebab1973d59a1fa9dfba1a65d674d477543b9aa33d3e32"} Oct 30 22:21:52.275310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.274943 5036 scope.go:122] "RemoveContainer" containerID="8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504" Oct 30 22:21:52.287665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.287646 5036 scope.go:122] "RemoveContainer" containerID="8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504" Oct 30 22:21:52.287960 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:21:52.287941 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504\": container with ID starting with 8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504 not found: ID does not exist" containerID="8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504" Oct 30 22:21:52.288015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.287970 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504"} err="failed to get container status \"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504\": rpc error: code = NotFound desc = could not find container \"8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504\": container with ID starting with 8ba8a7108862329229ed6098096a05ff0e024a32e30ac776045c1411c4145504 not found: ID does not exist" Oct 30 22:21:52.303464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.303437 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:52.309595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:52.309570 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876vqhlr"] Oct 30 22:21:54.191190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:54.191152 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e419b3-b295-47c1-bd79-5418afee5d9c" path="/var/lib/kubelet/pods/c2e419b3-b295-47c1-bd79-5418afee5d9c/volumes" Oct 30 22:21:58.031195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:58.031162 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9"] Oct 30 22:21:58.037638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:58.037609 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c996765f-qxlr9"] Oct 30 22:21:58.193717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:21:58.193683 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51108ee9-c2a3-4059-9329-9a3f770d9b22" path="/var/lib/kubelet/pods/51108ee9-c2a3-4059-9329-9a3f770d9b22/volumes" Oct 30 22:22:07.026231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:07.026194 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt"] Oct 30 22:22:07.031427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:07.031398 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2945cc1e-4msvt"] Oct 30 22:22:08.191157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:08.191120 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8db2bce-0087-4340-918f-d9e43bc8194e" path="/var/lib/kubelet/pods/e8db2bce-0087-4340-918f-d9e43bc8194e/volumes" Oct 30 22:22:18.101182 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:22:18.101103 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:22:20.541467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.541434 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:22:20.542345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542328 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c2e419b3-b295-47c1-bd79-5418afee5d9c" containerName="payload-processing" Oct 30 22:22:20.542395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542346 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e419b3-b295-47c1-bd79-5418afee5d9c" containerName="payload-processing" Oct 30 22:22:20.542395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542373 5036 cpu_manager.go:414] "containerMap: removing container" podUID="322a84c3-80e0-46bc-b6d2-b135f3171855" containerName="payload-processing" Oct 30 22:22:20.542395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542378 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322a84c3-80e0-46bc-b6d2-b135f3171855" containerName="payload-processing" Oct 30 22:22:20.542536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542524 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e419b3-b295-47c1-bd79-5418afee5d9c" containerName="payload-processing" Oct 30 22:22:20.542572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.542543 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="322a84c3-80e0-46bc-b6d2-b135f3171855" containerName="payload-processing" Oct 30 22:22:20.547461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.547438 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.553877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.553857 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-e6754ff0189c\"" Oct 30 22:22:20.563174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.563146 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:22:20.668152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.668103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.668545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.668518 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.669055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.669034 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbxf\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.669428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.669395 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.771132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.771099 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbxf\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.771356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.771181 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.771356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.771351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.771536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.771376 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.772047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.772017 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.772154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.772098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.774045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.774023 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.783612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.783586 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbxf\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.866603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.866521 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:20.916214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.916174 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:22:20.924526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.924488 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:20.932108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.931183 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:22:20.932295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:20.932159 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"agc-cred-f99017b412e9\"" Oct 30 22:22:21.038337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.035145 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7"] Oct 30 22:22:21.055166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.055125 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52cd56-6bpc7"] Oct 30 22:22:21.076690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.076602 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4jp\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.076893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.076725 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.076893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.076853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.077013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.076960 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.172901 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:22:21.172869 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c066c74_fcd2_4796_93f5_20640daf4bdc.slice/crio-dc1bf874687de3324346df7e9cba651731a3128bbbdb7acb73692fe4eab39ac9 WatchSource:0}: Error finding container dc1bf874687de3324346df7e9cba651731a3128bbbdb7acb73692fe4eab39ac9: Status 404 returned error can't find the container with id dc1bf874687de3324346df7e9cba651731a3128bbbdb7acb73692fe4eab39ac9 Oct 30 22:22:21.176322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.176276 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:22:21.178308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.178247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.178510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.178488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4jp\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.178680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.178583 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.178770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.178688 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.179132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.179093 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.179614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.179574 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.182181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.182158 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.190278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.190249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4jp\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.247127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.247104 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.606960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.606931 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:22:21.609084 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:22:21.609056 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234f8516_68cd_4216_8e00_fced9a946778.slice/crio-4574941b9a3830127733b869999e1f8c0acd71599dc68968ed41f5ca05a8e3fa WatchSource:0}: Error finding container 4574941b9a3830127733b869999e1f8c0acd71599dc68968ed41f5ca05a8e3fa: Status 404 returned error can't find the container with id 4574941b9a3830127733b869999e1f8c0acd71599dc68968ed41f5ca05a8e3fa Oct 30 22:22:21.856787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.856693 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" event={"ID":"1c066c74-fcd2-4796-93f5-20640daf4bdc","Type":"ContainerStarted","Data":"5a9ddfea7e675b3029b4e1e89b26ea6e15139aad72bf0d556e3dd2a4a53e40e8"} Oct 30 22:22:21.856787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.856758 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" event={"ID":"1c066c74-fcd2-4796-93f5-20640daf4bdc","Type":"ContainerStarted","Data":"dc1bf874687de3324346df7e9cba651731a3128bbbdb7acb73692fe4eab39ac9"} Oct 30 22:22:21.857020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.856797 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:21.858851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.858818 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" event={"ID":"234f8516-68cd-4216-8e00-fced9a946778","Type":"ContainerStarted","Data":"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea"} Oct 30 22:22:21.859007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.858873 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" event={"ID":"234f8516-68cd-4216-8e00-fced9a946778","Type":"ContainerStarted","Data":"4574941b9a3830127733b869999e1f8c0acd71599dc68968ed41f5ca05a8e3fa"} Oct 30 22:22:21.859007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.858905 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:21.881914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.881862 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" podStartSLOduration=1.88184458 podStartE2EDuration="1.88184458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:21.876681431 +0000 UTC m=+2565.826636379" watchObservedRunningTime="2026-10-30 22:22:21.88184458 +0000 UTC m=+2565.831799528" Oct 30 22:22:21.906859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:21.906814 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" podStartSLOduration=1.9068017290000001 podStartE2EDuration="1.906801729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:21.904307675 +0000 UTC m=+2565.854262635" watchObservedRunningTime="2026-10-30 22:22:21.906801729 +0000 UTC m=+2565.856756674" Oct 30 22:22:22.241095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:22.239005 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e424c15-66c3-42e3-b92e-729a99a87876" path="/var/lib/kubelet/pods/0e424c15-66c3-42e3-b92e-729a99a87876/volumes" Oct 30 22:22:27.030906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:27.030870 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b"] Oct 30 22:22:27.038340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:27.038315 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c739f8-t8z6b"] Oct 30 22:22:28.074752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:28.073826 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx"] Oct 30 22:22:28.095286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:28.095208 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2f674a-jh8lx"] Oct 30 22:22:28.191130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:28.191097 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9942370d-4276-4607-83c7-6fc1d20a1a9e" path="/var/lib/kubelet/pods/9942370d-4276-4607-83c7-6fc1d20a1a9e/volumes" Oct 30 22:22:28.191716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:28.191700 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1afbc47-0c4f-4b55-9053-d5dd35541e74" path="/var/lib/kubelet/pods/c1afbc47-0c4f-4b55-9053-d5dd35541e74/volumes" Oct 30 22:22:29.034991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:29.034949 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz"] Oct 30 22:22:29.043026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:29.042994 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30772e6-fqrkz"] Oct 30 22:22:30.191512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:30.191478 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a075d7-a1f9-4d51-bd30-51fb3e63a71c" path="/var/lib/kubelet/pods/c3a075d7-a1f9-4d51-bd30-51fb3e63a71c/volumes" Oct 30 22:22:32.873561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:32.873531 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:22:32.873985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:32.873608 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:22:32.939060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:32.938199 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:22:32.963175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:32.963143 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:22:35.095291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:35.094313 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c"] Oct 30 22:22:35.110516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:35.109285 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30332d58-7d57c"] Oct 30 22:22:36.193939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:36.193903 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89dd5d3-446d-47e4-a65a-8462a4730638" path="/var/lib/kubelet/pods/d89dd5d3-446d-47e4-a65a-8462a4730638/volumes" Oct 30 22:22:37.963574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:37.963526 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" podUID="808605bf-d216-480f-881d-641fd650b49d" containerName="payload-processing" containerID="cri-o://ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07" gracePeriod=25 Oct 30 22:22:37.984146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:37.984111 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" podUID="024ab905-32c6-4093-9149-989d38fb290c" containerName="payload-processing" containerID="cri-o://8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91" gracePeriod=25 Oct 30 22:22:38.358001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.357966 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:22:38.394583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.394556 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:22:38.453068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.450737 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\") pod \"808605bf-d216-480f-881d-641fd650b49d\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " Oct 30 22:22:38.453068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.451685 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume" pod "808605bf-d216-480f-881d-641fd650b49d" (UID: "808605bf-d216-480f-881d-641fd650b49d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:38.453068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.451829 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\") pod \"808605bf-d216-480f-881d-641fd650b49d\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " Oct 30 22:22:38.453655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.453375 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume" pod "808605bf-d216-480f-881d-641fd650b49d" (UID: "808605bf-d216-480f-881d-641fd650b49d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:38.453655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.453437 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\") pod \"808605bf-d216-480f-881d-641fd650b49d\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " Oct 30 22:22:38.453655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.453610 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\") pod \"808605bf-d216-480f-881d-641fd650b49d\" (UID: \"808605bf-d216-480f-881d-641fd650b49d\") " Oct 30 22:22:38.455092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.455070 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.455092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.455092 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/808605bf-d216-480f-881d-641fd650b49d-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.457625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.457594 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4" pod "808605bf-d216-480f-881d-641fd650b49d" (UID: "808605bf-d216-480f-881d-641fd650b49d"). InnerVolumeSpecName "kube-api-access-vmmw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:38.457740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.457599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials" pod "808605bf-d216-480f-881d-641fd650b49d" (UID: "808605bf-d216-480f-881d-641fd650b49d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:38.556708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.556615 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\") pod \"024ab905-32c6-4093-9149-989d38fb290c\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " Oct 30 22:22:38.556896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.556866 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\") pod \"024ab905-32c6-4093-9149-989d38fb290c\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " Oct 30 22:22:38.557001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.556985 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\") pod \"024ab905-32c6-4093-9149-989d38fb290c\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " Oct 30 22:22:38.557077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557035 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\") pod \"024ab905-32c6-4093-9149-989d38fb290c\" (UID: \"024ab905-32c6-4093-9149-989d38fb290c\") " Oct 30 22:22:38.557077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume" pod "024ab905-32c6-4093-9149-989d38fb290c" (UID: "024ab905-32c6-4093-9149-989d38fb290c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:38.557481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557451 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume" pod "024ab905-32c6-4093-9149-989d38fb290c" (UID: "024ab905-32c6-4093-9149-989d38fb290c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:38.557858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557840 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.557949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmw4\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-kube-api-access-vmmw4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.557949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557881 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/808605bf-d216-480f-881d-641fd650b49d-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.557949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.557895 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024ab905-32c6-4093-9149-989d38fb290c-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.559992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.559967 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials" pod "024ab905-32c6-4093-9149-989d38fb290c" (UID: "024ab905-32c6-4093-9149-989d38fb290c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:38.560090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.560006 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp" pod "024ab905-32c6-4093-9149-989d38fb290c" (UID: "024ab905-32c6-4093-9149-989d38fb290c"). InnerVolumeSpecName "kube-api-access-xvlzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:38.659769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.659734 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlzp\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-kube-api-access-xvlzp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:38.659769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:38.659767 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/024ab905-32c6-4093-9149-989d38fb290c-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:39.036290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.033966 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd"] Oct 30 22:22:39.044242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.044212 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt"] Oct 30 22:22:39.052255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.052217 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11073557-kgdhd"] Oct 30 22:22:39.053412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.053385 5036 generic.go:356] "Generic (PLEG): container finished" podID="808605bf-d216-480f-881d-641fd650b49d" containerID="ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07" exitCode=0 Oct 30 22:22:39.053550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.053495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" event={"ID":"808605bf-d216-480f-881d-641fd650b49d","Type":"ContainerDied","Data":"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07"} Oct 30 22:22:39.053550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.053541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" event={"ID":"808605bf-d216-480f-881d-641fd650b49d","Type":"ContainerDied","Data":"dff962ac229a7db0cdc406aff35f03dc081bdf2ceae16f07991fcec36ba96346"} Oct 30 22:22:39.053733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.053567 5036 scope.go:122] "RemoveContainer" containerID="ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07" Oct 30 22:22:39.053733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.053592 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6" Oct 30 22:22:39.059818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.059794 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3109ca2f-8tczt"] Oct 30 22:22:39.060418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.060392 5036 generic.go:356] "Generic (PLEG): container finished" podID="024ab905-32c6-4093-9149-989d38fb290c" containerID="8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91" exitCode=0 Oct 30 22:22:39.060549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.060446 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" event={"ID":"024ab905-32c6-4093-9149-989d38fb290c","Type":"ContainerDied","Data":"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91"} Oct 30 22:22:39.060549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.060484 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" event={"ID":"024ab905-32c6-4093-9149-989d38fb290c","Type":"ContainerDied","Data":"4e408fa2442127492815cf26f460b059fcfcdf1c027c41db0b4a2f03f29d528b"} Oct 30 22:22:39.060662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.060554 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96" Oct 30 22:22:39.072106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.072083 5036 scope.go:122] "RemoveContainer" containerID="ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07" Oct 30 22:22:39.072419 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:22:39.072402 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07\": container with ID starting with ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07 not found: ID does not exist" containerID="ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07" Oct 30 22:22:39.072511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.072425 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07"} err="failed to get container status \"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07\": rpc error: code = NotFound desc = could not find container \"ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07\": container with ID starting with ae0f7f2dd4196fee900586daf75c997f7c14cf6d514e0d8437ff971ba4711a07 not found: ID does not exist" Oct 30 22:22:39.072511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.072452 5036 scope.go:122] "RemoveContainer" containerID="8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91" Oct 30 22:22:39.084601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.084576 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:22:39.084729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.084655 5036 scope.go:122] "RemoveContainer" containerID="8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91" Oct 30 22:22:39.085003 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:22:39.084978 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91\": container with ID starting with 8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91 not found: ID does not exist" containerID="8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91" Oct 30 22:22:39.085089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.085007 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91"} err="failed to get container status \"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91\": rpc error: code = NotFound desc = could not find container \"8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91\": container with ID starting with 8a05bcb9fd58e98a8738555c551bb19edce90c5a9f1145be9412fb6b876a7c91 not found: ID does not exist" Oct 30 22:22:39.090008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.089989 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6zwlm6"] Oct 30 22:22:39.098750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.098715 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:22:39.103277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:39.103245 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-78775bh96"] Oct 30 22:22:40.194933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:40.194897 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024ab905-32c6-4093-9149-989d38fb290c" path="/var/lib/kubelet/pods/024ab905-32c6-4093-9149-989d38fb290c/volumes" Oct 30 22:22:40.195606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:40.195581 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4228e118-12fe-4495-83de-10dd93af169b" path="/var/lib/kubelet/pods/4228e118-12fe-4495-83de-10dd93af169b/volumes" Oct 30 22:22:40.196174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:40.196157 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d826f90-f0f0-497f-92d7-93524eb290c5" path="/var/lib/kubelet/pods/7d826f90-f0f0-497f-92d7-93524eb290c5/volumes" Oct 30 22:22:40.196961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:40.196944 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808605bf-d216-480f-881d-641fd650b49d" path="/var/lib/kubelet/pods/808605bf-d216-480f-881d-641fd650b49d/volumes" Oct 30 22:22:48.106045 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:22:48.106000 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:22:48.553439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:48.553406 5036 scope.go:122] "RemoveContainer" containerID="ffafdfa8379e1f4e4dbd80e9a251173b14b34af37cb06053aea3df73a888570a" Oct 30 22:22:48.565874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:48.565851 5036 scope.go:122] "RemoveContainer" containerID="3b849766b583a273985dd972b4847c98ffe8c9a7e4ed5044e7201ad13fd4f16c" Oct 30 22:22:52.038865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:52.038819 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6"] Oct 30 22:22:52.044196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:52.044160 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cdb0ea5-tpkd6"] Oct 30 22:22:52.195283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:52.195238 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a38797-5d90-411a-b8bd-1eace3fd6eb2" path="/var/lib/kubelet/pods/49a38797-5d90-411a-b8bd-1eace3fd6eb2/volumes" Oct 30 22:22:53.032165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:53.032135 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl"] Oct 30 22:22:53.040688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:53.040652 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab8bd72-srfkl"] Oct 30 22:22:54.191309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:22:54.191251 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfe6766-bd59-4b96-89fe-b0829a695155" path="/var/lib/kubelet/pods/ebfe6766-bd59-4b96-89fe-b0829a695155/volumes" Oct 30 22:23:18.100681 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:18.100642 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:23:25.616121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.616086 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2"] Oct 30 22:23:25.616937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.616919 5036 cpu_manager.go:414] "containerMap: removing container" podUID="808605bf-d216-480f-881d-641fd650b49d" containerName="payload-processing" Oct 30 22:23:25.616937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.616937 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808605bf-d216-480f-881d-641fd650b49d" containerName="payload-processing" Oct 30 22:23:25.617083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.616956 5036 cpu_manager.go:414] "containerMap: removing container" podUID="024ab905-32c6-4093-9149-989d38fb290c" containerName="payload-processing" Oct 30 22:23:25.617083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.616963 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024ab905-32c6-4093-9149-989d38fb290c" containerName="payload-processing" Oct 30 22:23:25.617199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.617136 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="808605bf-d216-480f-881d-641fd650b49d" containerName="payload-processing" Oct 30 22:23:25.617199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.617147 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="024ab905-32c6-4093-9149-989d38fb290c" containerName="payload-processing" Oct 30 22:23:25.621312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.621291 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.627083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.627058 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:23:25.627226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.627134 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:25.627226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.627159 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:25.628200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.628180 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:25.633394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.633372 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2"] Oct 30 22:23:25.733415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.733367 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dvsc\" (UniqueName: \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.733761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.733737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.733885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.733863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.835751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.835669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.835958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.835761 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.835958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.835898 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dvsc\" (UniqueName: \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.836684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.836648 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.838450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.838429 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.847895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.847866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dvsc\" (UniqueName: \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\") pod \"maas-api-revoke-sub-6961d28c-mh2x2\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:25.959291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:25.944249 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:26.284231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:26.284200 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2"] Oct 30 22:23:26.284858 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:26.284834 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e05283_26cb_4592_95aa_3586d17c19bf.slice/crio-3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0 WatchSource:0}: Error finding container 3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0: Status 404 returned error can't find the container with id 3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0 Oct 30 22:23:26.687935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:26.687900 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" event={"ID":"50e05283-26cb-4592-95aa-3586d17c19bf","Type":"ContainerStarted","Data":"3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0"} Oct 30 22:23:26.841993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:26.841957 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:23:26.842255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:26.842209 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" containerID="cri-o://5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801" gracePeriod=30 Oct 30 22:23:27.698581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:27.698549 5036 generic.go:356] "Generic (PLEG): container finished" podID="50e05283-26cb-4592-95aa-3586d17c19bf" containerID="a2aba00e7c37621986b03c8727642863720d4ed45f757a9e7d0a8a0eccc1525b" exitCode=0 Oct 30 22:23:27.698991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:27.698631 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" event={"ID":"50e05283-26cb-4592-95aa-3586d17c19bf","Type":"ContainerDied","Data":"a2aba00e7c37621986b03c8727642863720d4ed45f757a9e7d0a8a0eccc1525b"} Oct 30 22:23:28.902109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:28.902080 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:29.004845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.004748 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\" (UniqueName: \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\") pod \"50e05283-26cb-4592-95aa-3586d17c19bf\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " Oct 30 22:23:29.005503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.005475 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\") pod \"50e05283-26cb-4592-95aa-3586d17c19bf\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " Oct 30 22:23:29.006227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.006033 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca" pod "50e05283-26cb-4592-95aa-3586d17c19bf" (UID: "50e05283-26cb-4592-95aa-3586d17c19bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:29.006646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.006617 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\") pod \"50e05283-26cb-4592-95aa-3586d17c19bf\" (UID: \"50e05283-26cb-4592-95aa-3586d17c19bf\") " Oct 30 22:23:29.007931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.007902 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.010124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.010099 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc" pod "50e05283-26cb-4592-95aa-3586d17c19bf" (UID: "50e05283-26cb-4592-95aa-3586d17c19bf"). InnerVolumeSpecName "kube-api-access-8dvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:29.027509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.027480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca" pod "50e05283-26cb-4592-95aa-3586d17c19bf" (UID: "50e05283-26cb-4592-95aa-3586d17c19bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:29.109336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.109300 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e05283-26cb-4592-95aa-3586d17c19bf-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.109336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.109334 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dvsc\" (UniqueName: \"kubernetes.io/projected/50e05283-26cb-4592-95aa-3586d17c19bf-kube-api-access-8dvsc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.721135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.721100 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" Oct 30 22:23:29.721135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.721115 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2" event={"ID":"50e05283-26cb-4592-95aa-3586d17c19bf","Type":"ContainerDied","Data":"3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0"} Oct 30 22:23:29.721385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:29.721174 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb3a959d8a59aa49d17ad2f8a6c6353fa0627116be4a5e4b1968651130ae8f0" Oct 30 22:23:32.157867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.157835 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.306912 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307098 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307361 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307761 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.309300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.307883 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\") pod \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\" (UID: \"4cc5fa70-0515-4c71-9818-2e86b6b8402b\") " Oct 30 22:23:32.324481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.323441 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:32.324481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.323652 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:32.324481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.324148 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:32.354321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.331593 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:32.354321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.334602 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:32.370727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.370636 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "kube-api-access-bbf9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:32.371083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.371054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:32.371222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.371102 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:32.373339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.371969 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy" pod "4cc5fa70-0515-4c71-9818-2e86b6b8402b" (UID: "4cc5fa70-0515-4c71-9818-2e86b6b8402b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:32.411164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411129 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411160 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411171 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411180 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411188 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411196 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411205 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf9t\" (UniqueName: \"kubernetes.io/projected/4cc5fa70-0515-4c71-9818-2e86b6b8402b-kube-api-access-bbf9t\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411215 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.411421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.411223 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc5fa70-0515-4c71-9818-2e86b6b8402b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.756842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.756803 5036 generic.go:356] "Generic (PLEG): container finished" podID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerID="5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801" exitCode=0 Oct 30 22:23:32.757054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.756944 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" Oct 30 22:23:32.757054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.756944 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerDied","Data":"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801"} Oct 30 22:23:32.757181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.757107 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9" event={"ID":"4cc5fa70-0515-4c71-9818-2e86b6b8402b","Type":"ContainerDied","Data":"2ae43212ae5ea2c18d7eda948680861707fe1588aec53bc2acf471e2447a97f7"} Oct 30 22:23:32.757181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.757131 5036 scope.go:122] "RemoveContainer" containerID="5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801" Oct 30 22:23:32.770907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.770889 5036 scope.go:122] "RemoveContainer" containerID="b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b" Oct 30 22:23:32.782388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.782363 5036 scope.go:122] "RemoveContainer" containerID="5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801" Oct 30 22:23:32.782748 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:32.782723 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801\": container with ID starting with 5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801 not found: ID does not exist" containerID="5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801" Oct 30 22:23:32.782902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.782761 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801"} err="failed to get container status \"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801\": rpc error: code = NotFound desc = could not find container \"5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801\": container with ID starting with 5aa7cacc71eb3ed4e60897df6779f9c3c2682b937364eecd4b3be0d4844c6801 not found: ID does not exist" Oct 30 22:23:32.782902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.782785 5036 scope.go:122] "RemoveContainer" containerID="b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b" Oct 30 22:23:32.783121 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:32.783101 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b\": container with ID starting with b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b not found: ID does not exist" containerID="b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b" Oct 30 22:23:32.783163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.783127 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b"} err="failed to get container status \"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b\": rpc error: code = NotFound desc = could not find container \"b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b\": container with ID starting with b54ebf659a079530932883193040231b7bb6e1f806fae3a55839654c5c3cab8b not found: ID does not exist" Oct 30 22:23:32.784893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.784872 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:23:32.791904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:32.791885 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-qpkh9"] Oct 30 22:23:34.215838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:34.215797 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" path="/var/lib/kubelet/pods/4cc5fa70-0515-4c71-9818-2e86b6b8402b/volumes" Oct 30 22:23:37.078514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.078482 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:23:37.078915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.078682 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" containerID="cri-o://1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384" gracePeriod=30 Oct 30 22:23:37.123337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.123287 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:23:37.124180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124161 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:37.124307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124180 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:37.124307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124213 5036 cpu_manager.go:414] "containerMap: removing container" podUID="50e05283-26cb-4592-95aa-3586d17c19bf" containerName="revoke-keys" Oct 30 22:23:37.124307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124219 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e05283-26cb-4592-95aa-3586d17c19bf" containerName="revoke-keys" Oct 30 22:23:37.124307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124252 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:37.124307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124257 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:37.124589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124420 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:37.124589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.124443 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e05283-26cb-4592-95aa-3586d17c19bf" containerName="revoke-keys" Oct 30 22:23:37.130328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.130302 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.132731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.132703 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:23:37.193655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.193619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.193816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.193679 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.193901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.193872 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.194017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.194000 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r5d\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.321608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.321564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.337497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.323630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.337769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.337597 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.338207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.338180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.339450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.338490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r5d\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.339450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.339016 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.355774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.354935 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.363293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.361973 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r5d\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\") pod \"payload-processing-external-model-7698d5fd68-bzqjx\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.450568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.450534 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:37.924705 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:37.924666 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ac6157_31df_48a0_80c1_edaa6e4d59d0.slice/crio-5012b5337e8319b1a4ca3a4f482fda3978d51df69def259ba1c0d774921cd095 WatchSource:0}: Error finding container 5012b5337e8319b1a4ca3a4f482fda3978d51df69def259ba1c0d774921cd095: Status 404 returned error can't find the container with id 5012b5337e8319b1a4ca3a4f482fda3978d51df69def259ba1c0d774921cd095 Oct 30 22:23:37.924844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:37.924820 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:23:38.856471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:38.856424 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" event={"ID":"62ac6157-31df-48a0-80c1-edaa6e4d59d0","Type":"ContainerStarted","Data":"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356"} Oct 30 22:23:38.856950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:38.856515 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" event={"ID":"62ac6157-31df-48a0-80c1-edaa6e4d59d0","Type":"ContainerStarted","Data":"5012b5337e8319b1a4ca3a4f482fda3978d51df69def259ba1c0d774921cd095"} Oct 30 22:23:38.856950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:38.856546 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:38.880850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:38.880791 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" podStartSLOduration=1.880773713 podStartE2EDuration="1.880773713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:38.874502224 +0000 UTC m=+2642.824457171" watchObservedRunningTime="2026-10-30 22:23:38.880773713 +0000 UTC m=+2642.830728660" Oct 30 22:23:41.842760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.842724 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:41.843817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.843797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc5fa70-0515-4c71-9818-2e86b6b8402b" containerName="istio-proxy" Oct 30 22:23:41.848097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.848079 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:41.855314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.853304 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1a5bdf\"/\"agc-cred-20b43f7a377a\"" Oct 30 22:23:41.859732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.859706 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:41.969633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.969597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:41.969806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.969736 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:41.969887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.969815 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:41.969994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:41.969968 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6z5\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.071239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.071199 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.072087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.072059 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.073221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.073201 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.074100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.074077 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6z5\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.081650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.073940 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.081858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.073080 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.118070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.117988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.141918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.141879 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6z5\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.170852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.170820 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:42.208531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.208503 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:42.214319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.213551 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.220700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.220658 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-72c60152b5ed\"" Oct 30 22:23:42.245861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.245495 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:42.285081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.285041 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.285249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.285145 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.285361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.285296 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.285361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.285348 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.426998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.426647 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.427224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.427194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.427385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.427363 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.427489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.427422 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.428467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.428399 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.428965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.428943 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.435328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.434017 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.441246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.441224 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.472293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.472256 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:23:42.486371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.486348 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:42.488794 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:42.488770 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1c62f5_aeef_4c2b_a347_4c0a1dc59cf7.slice/crio-58df1eeaab43a677ead8ed6465e6f4c3a44f3e7afebd2f422597a80cc64d64a7 WatchSource:0}: Error finding container 58df1eeaab43a677ead8ed6465e6f4c3a44f3e7afebd2f422597a80cc64d64a7: Status 404 returned error can't find the container with id 58df1eeaab43a677ead8ed6465e6f4c3a44f3e7afebd2f422597a80cc64d64a7 Oct 30 22:23:42.569961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.569936 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:42.631884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.631848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.631926 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.631995 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632017 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632057 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632141 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632172 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.632401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.632199 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\") pod \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\" (UID: \"31b289c4-9657-4f1e-83e9-2b8a91c40af0\") " Oct 30 22:23:42.634129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.633686 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.634129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.633930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.634434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.634298 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:42.635523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.635234 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.635811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.635137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.636554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.636526 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.636669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.636559 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.636669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.636578 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.636669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.636594 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.636669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.636611 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.637424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.637381 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.637548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.637461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.637913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.637892 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "kube-api-access-2clqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.638132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.638111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo" pod "31b289c4-9657-4f1e-83e9-2b8a91c40af0" (UID: "31b289c4-9657-4f1e-83e9-2b8a91c40af0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:42.738522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.738431 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.738522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.738465 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clqj\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-kube-api-access-2clqj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.738522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.738477 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.738522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.738490 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b289c4-9657-4f1e-83e9-2b8a91c40af0-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.894143 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:42.894110 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839f5ad2_8a45_45ae_a7e8_65ac3751edf3.slice/crio-843a66e381590956a1587597396a8232cf1d60c6c0d509e6e6137aa38484aaf6 WatchSource:0}: Error finding container 843a66e381590956a1587597396a8232cf1d60c6c0d509e6e6137aa38484aaf6: Status 404 returned error can't find the container with id 843a66e381590956a1587597396a8232cf1d60c6c0d509e6e6137aa38484aaf6 Oct 30 22:23:42.895426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.895400 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:43.002347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.996198 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" event={"ID":"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7","Type":"ContainerStarted","Data":"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc"} Oct 30 22:23:43.002347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.996341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" event={"ID":"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7","Type":"ContainerStarted","Data":"58df1eeaab43a677ead8ed6465e6f4c3a44f3e7afebd2f422597a80cc64d64a7"} Oct 30 22:23:43.002347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:42.998054 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:43.003480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.003449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" event={"ID":"839f5ad2-8a45-45ae-a7e8-65ac3751edf3","Type":"ContainerStarted","Data":"843a66e381590956a1587597396a8232cf1d60c6c0d509e6e6137aa38484aaf6"} Oct 30 22:23:43.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.021958 5036 generic.go:356] "Generic (PLEG): container finished" podID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerID="1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384" exitCode=0 Oct 30 22:23:43.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.022162 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" event={"ID":"31b289c4-9657-4f1e-83e9-2b8a91c40af0","Type":"ContainerDied","Data":"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384"} Oct 30 22:23:43.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.022295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" event={"ID":"31b289c4-9657-4f1e-83e9-2b8a91c40af0","Type":"ContainerDied","Data":"e78955b27bd0903249574aa973693c260e4a8acb0a770c90f2fc00cb39643a65"} Oct 30 22:23:43.022358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.022342 5036 scope.go:122] "RemoveContainer" containerID="1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384" Oct 30 22:23:43.022783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.022590 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55" Oct 30 22:23:43.023377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.023329 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" podStartSLOduration=2.023302625 podStartE2EDuration="2.023302625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:43.021194082 +0000 UTC m=+2646.971149046" watchObservedRunningTime="2026-10-30 22:23:43.023302625 +0000 UTC m=+2646.973257571" Oct 30 22:23:43.048720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.048691 5036 scope.go:122] "RemoveContainer" containerID="1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384" Oct 30 22:23:43.049368 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:43.049340 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384\": container with ID starting with 1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384 not found: ID does not exist" containerID="1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384" Oct 30 22:23:43.049501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.049374 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384"} err="failed to get container status \"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384\": rpc error: code = NotFound desc = could not find container \"1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384\": container with ID starting with 1e53d2d9964ed8f2f9dbedc512d0ce0333a35e243cc509ce3d6a62425f27e384 not found: ID does not exist" Oct 30 22:23:43.058804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.058779 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:23:43.065636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:43.065611 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-9qq55"] Oct 30 22:23:44.035321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.035284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" event={"ID":"839f5ad2-8a45-45ae-a7e8-65ac3751edf3","Type":"ContainerStarted","Data":"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941"} Oct 30 22:23:44.035924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.035893 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:44.058501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.058453 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" podStartSLOduration=2.058440808 podStartE2EDuration="2.058440808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:44.053335465 +0000 UTC m=+2648.003290405" watchObservedRunningTime="2026-10-30 22:23:44.058440808 +0000 UTC m=+2648.008395779" Oct 30 22:23:44.192083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.192040 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" path="/var/lib/kubelet/pods/31b289c4-9657-4f1e-83e9-2b8a91c40af0/volumes" Oct 30 22:23:44.325156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.325081 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 22:23:44.325336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:44.325259 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" containerID="cri-o://0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8" gracePeriod=30 Oct 30 22:23:48.100463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:48.100375 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:23:48.674292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.666020 5036 scope.go:122] "RemoveContainer" containerID="6c7c86d6fbb55ae402f9c04384332cbcf17d6f08fb28880e4c371426eb24241c" Oct 30 22:23:48.760997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.758354 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:48.791237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.791205 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:48.793311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.793291 5036 scope.go:122] "RemoveContainer" containerID="28a018f7e741015bb3d67541d4cc0a8e772b01de29452a932c11f9dcfc4ae1b6" Oct 30 22:23:48.808805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.808779 5036 scope.go:122] "RemoveContainer" containerID="cb5b1abeff7eea4fbfa0f06ca7f9f98c29221493fc2583acb63012a26eae8c19" Oct 30 22:23:48.821220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.821198 5036 scope.go:122] "RemoveContainer" containerID="95b5c8d2da6bee854dfd7f6b56299a443cfcb2919b8378acfcbefbf0f9b25e74" Oct 30 22:23:48.842122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.842099 5036 scope.go:122] "RemoveContainer" containerID="81a76cab1b841e3fc9881df34cb79f74dd2b936113733b7e311fd1671e2b8bf7" Oct 30 22:23:48.852914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.852886 5036 scope.go:122] "RemoveContainer" containerID="868b46f52c4dc969a812aa6605e49367b47662ae80b0b0554e90d6bfa15a6aa5" Oct 30 22:23:48.863483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.863466 5036 scope.go:122] "RemoveContainer" containerID="226557496bcdeefa580e95450121a3e3003f6ac2ce041331f06b150cf67a265e" Oct 30 22:23:48.873434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.873414 5036 scope.go:122] "RemoveContainer" containerID="ba7f4b8aa7f66b09141793be392100df6247d268dd10d4f106bd2c4884ee9ebb" Oct 30 22:23:48.883656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:48.883637 5036 scope.go:122] "RemoveContainer" containerID="c23dcf5b60cfde1e702135a0ecd7ad40a6253f16e3e5684fef06dab2ecbfdc5a" Oct 30 22:23:49.290424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.290335 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc"] Oct 30 22:23:49.303741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.292225 5036 cpu_manager.go:414] "containerMap: removing container" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" Oct 30 22:23:49.307299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.304423 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" Oct 30 22:23:49.307299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.304837 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b289c4-9657-4f1e-83e9-2b8a91c40af0" containerName="istio-proxy" Oct 30 22:23:49.377615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.365978 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.377615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.374824 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:49.377615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.375536 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:49.377615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.375647 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:23:49.377615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.375710 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:49.406251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.406214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hct\" (UniqueName: \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.406599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.406577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.407071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.407050 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.429300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.418978 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc"] Oct 30 22:23:49.562811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.562701 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hct\" (UniqueName: \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.563080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.563062 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.563421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.563399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.590689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.590656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.732503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.732470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.735902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.735870 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hct\" (UniqueName: \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\") pod \"maas-api-revoke-sub-a1c574e7-2jdrc\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.740660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.740637 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:49.747354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.747337 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:49.801464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.801395 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:49.894224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.890069 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:23:49.918717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.918388 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 22:23:49.959315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:49.952055 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:23:50.023889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.023852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.023979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024016 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024098 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024132 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024210 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024249 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024356 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024521 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\") pod \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\" (UID: \"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae\") " Oct 30 22:23:50.024783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:50.024974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.024953 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:50.025307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025064 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:50.025307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:50.025521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025499 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.025597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025528 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.025597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025544 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.025597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025562 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.025967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.025944 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:50.027353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.027328 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:50.027465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.027375 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:50.027801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.027777 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "kube-api-access-ft7jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:50.028888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.028868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo" pod "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" (UID: "4e6939fb-1bd2-40c9-b53a-0d0454dc69ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:50.127820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.127785 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7jv\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-kube-api-access-ft7jv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.127939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.127826 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.127939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.127843 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.127939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.127856 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.127939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.127868 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.129854 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:50.129827 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d70e4d_8e79_4042_9bea_28a4f4fbdf1a.slice/crio-25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f WatchSource:0}: Error finding container 25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f: Status 404 returned error can't find the container with id 25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f Oct 30 22:23:50.132699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.132673 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc"] Oct 30 22:23:50.404119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.404084 5036 generic.go:356] "Generic (PLEG): container finished" podID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerID="0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8" exitCode=0 Oct 30 22:23:50.404647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.404158 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" Oct 30 22:23:50.405386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.404177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" event={"ID":"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae","Type":"ContainerDied","Data":"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8"} Oct 30 22:23:50.405557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.405404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6" event={"ID":"4e6939fb-1bd2-40c9-b53a-0d0454dc69ae","Type":"ContainerDied","Data":"0c287a466388f092e9e6cd865c9ac0779c9c9fa49324dfaafe2ce7c59270debd"} Oct 30 22:23:50.405557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.405434 5036 scope.go:122] "RemoveContainer" containerID="0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8" Oct 30 22:23:50.409056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.408868 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" event={"ID":"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a","Type":"ContainerStarted","Data":"25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f"} Oct 30 22:23:50.422602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.422585 5036 scope.go:122] "RemoveContainer" containerID="0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8" Oct 30 22:23:50.422897 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:50.422876 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8\": container with ID starting with 0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8 not found: ID does not exist" containerID="0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8" Oct 30 22:23:50.422979 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.422904 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8"} err="failed to get container status \"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8\": rpc error: code = NotFound desc = could not find container \"0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8\": container with ID starting with 0efba816aaf45a11fcdbb898b2a5a466dfe8c2d93dfc9776ce9bd41c2debb1a8 not found: ID does not exist" Oct 30 22:23:50.430521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.430250 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 22:23:50.437948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.437926 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-djkk6"] Oct 30 22:23:50.565972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.565936 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:23:50.566225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.566165 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" containerID="cri-o://22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04" gracePeriod=30 Oct 30 22:23:50.573399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.573366 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:23:50.573616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:50.573595 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" containerID="cri-o://e8140be39060ee801ea85ad496afd715865e5485efc02e81849181e889c828e7" gracePeriod=30 Oct 30 22:23:51.492217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.492177 5036 generic.go:356] "Generic (PLEG): container finished" podID="67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" containerID="ec5f6348203296661616394277ea513a6e22d45a56c9c36e7cc74e5524f92d59" exitCode=0 Oct 30 22:23:51.492715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.492370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" event={"ID":"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a","Type":"ContainerDied","Data":"ec5f6348203296661616394277ea513a6e22d45a56c9c36e7cc74e5524f92d59"} Oct 30 22:23:51.800609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.800574 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:23:51.801426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.801409 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" Oct 30 22:23:51.801498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.801427 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" Oct 30 22:23:51.801609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.801599 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" containerName="istio-proxy" Oct 30 22:23:51.805654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.805636 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:51.815616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.815593 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"openshift-service-ca.crt\"" Oct 30 22:23:51.815854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.815621 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"payload-processing-external-model-e2e--ac81303a-dockercfg-vmd26\"" Oct 30 22:23:51.815854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.815770 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a39158\"" Oct 30 22:23:51.816052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.815869 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"routing-overlay\"" Oct 30 22:23:51.816052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.816032 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"kube-root-ca.crt\"" Oct 30 22:23:51.816202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.816099 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a39158\"/\"agc-cred-e07d07bd5a0f\"" Oct 30 22:23:51.818244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.818214 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:23:51.936290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.928418 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:51.936290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.928552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:51.936290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.928730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxcb\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:51.936290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:51.928837 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.042340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.041439 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.042340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.041634 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxcb\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.042340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.041735 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.042340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.041869 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.046289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.042887 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.046289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.043497 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.063338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.061162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.091288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.085977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxcb\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\") pod \"payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.135292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.132574 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:52.239362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.238687 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6939fb-1bd2-40c9-b53a-0d0454dc69ae" path="/var/lib/kubelet/pods/4e6939fb-1bd2-40c9-b53a-0d0454dc69ae/volumes" Oct 30 22:23:52.386822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.386792 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:23:52.393050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.393026 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.397038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397011 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"kube-root-ca.crt\"" Oct 30 22:23:52.397182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397050 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"routing-overlay\"" Oct 30 22:23:52.397244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397204 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"openshift-service-ca.crt\"" Oct 30 22:23:52.397318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397238 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-e2e--a9381736-dockercfg-6mk48\"" Oct 30 22:23:52.397373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397315 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-837a279b6a0e\"" Oct 30 22:23:52.397373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.397329 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-plugins-e2e-extmod-b-26a128\"" Oct 30 22:23:52.402898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.402876 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:23:52.456291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.451770 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zwp\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.456291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.451881 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.456291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.451914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.456291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.451978 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.614920 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.614983 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.620292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.615056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.811121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.636858 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.811121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.646907 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zwp\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.811121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.657639 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.829454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.817066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.849120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.848981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zwp\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:52.854655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:52.854628 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:23:53.037194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.037169 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:53.041026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.041002 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:53.161974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.161937 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\") pod \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " Oct 30 22:23:53.162424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.162397 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\") pod \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " Oct 30 22:23:53.162547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.162524 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\" (UniqueName: \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\") pod \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\" (UID: \"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a\") " Oct 30 22:23:53.162918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.162889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca" pod "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" (UID: "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:53.163381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.163358 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:53.165104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.165074 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca" pod "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" (UID: "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:53.165801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.165764 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct" pod "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" (UID: "67d70e4d-8e79-4042-9bea-28a4f4fbdf1a"). InnerVolumeSpecName "kube-api-access-d5hct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:53.265283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.265223 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hct\" (UniqueName: \"kubernetes.io/projected/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-kube-api-access-d5hct\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:53.265283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.265255 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:53.414461 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:23:53.414419 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ed73a5_8dbb_455b_9a2d_44bb0aa04820.slice/crio-6ed7305786899cb0891d85436745af55d83e255d40b49e428e914534176401b8 WatchSource:0}: Error finding container 6ed7305786899cb0891d85436745af55d83e255d40b49e428e914534176401b8: Status 404 returned error can't find the container with id 6ed7305786899cb0891d85436745af55d83e255d40b49e428e914534176401b8 Oct 30 22:23:53.417182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.417159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:23:53.759005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.758882 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" podUID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" containerName="payload-processing" containerID="cri-o://0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc" gracePeriod=25 Oct 30 22:23:53.862559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.862529 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" Oct 30 22:23:53.862753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.862538 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc" event={"ID":"67d70e4d-8e79-4042-9bea-28a4f4fbdf1a","Type":"ContainerDied","Data":"25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f"} Oct 30 22:23:53.862753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.862658 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ece995c5de559ace975527a3d62d1e307dd83de36fb51a40fc089655a5182f" Oct 30 22:23:53.864543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.864514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" event={"ID":"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820","Type":"ContainerStarted","Data":"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0"} Oct 30 22:23:53.864671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.864572 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" event={"ID":"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820","Type":"ContainerStarted","Data":"6ed7305786899cb0891d85436745af55d83e255d40b49e428e914534176401b8"} Oct 30 22:23:53.864671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.864600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:23:53.866396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.866373 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" event={"ID":"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb","Type":"ContainerStarted","Data":"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8"} Oct 30 22:23:53.866512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.866415 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" event={"ID":"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb","Type":"ContainerStarted","Data":"c40f13ffecf3e702f0aa3f439920508360582f13b23400b0d5ca437452a355fb"} Oct 30 22:23:53.866512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.866466 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:23:53.884474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.884417 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" podStartSLOduration=1.884400242 podStartE2EDuration="1.884400242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:53.882694747 +0000 UTC m=+2657.832649693" watchObservedRunningTime="2026-10-30 22:23:53.884400242 +0000 UTC m=+2657.834355188" Oct 30 22:23:53.903674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:53.903630 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" podStartSLOduration=2.903616447 podStartE2EDuration="2.903616447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:53.901641383 +0000 UTC m=+2657.851596341" watchObservedRunningTime="2026-10-30 22:23:53.903616447 +0000 UTC m=+2657.853571392" Oct 30 22:23:54.057754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.057728 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:54.190987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.190957 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\") pod \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " Oct 30 22:23:54.191158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.191131 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\") pod \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " Oct 30 22:23:54.191251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.191240 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\") pod \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " Oct 30 22:23:54.191429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.191413 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\") pod \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\" (UID: \"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7\") " Oct 30 22:23:54.191627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.191587 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume" pod "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" (UID: "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:54.209307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.191842 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume" pod "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" (UID: "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:54.209307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.192455 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:54.209307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.192481 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:54.209307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.193986 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5" pod "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" (UID: "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7"). InnerVolumeSpecName "kube-api-access-dm6z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:54.209307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.195427 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials" pod "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" (UID: "2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:54.294792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.294710 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6z5\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-kube-api-access-dm6z5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:54.294792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.294733 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:54.758390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.757572 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" podUID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" containerName="payload-processing" containerID="cri-o://43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941" gracePeriod=25 Oct 30 22:23:54.876903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.876867 5036 generic.go:356] "Generic (PLEG): container finished" podID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" containerID="0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc" exitCode=0 Oct 30 22:23:54.877378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.876920 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" Oct 30 22:23:54.877378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.876944 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" event={"ID":"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7","Type":"ContainerDied","Data":"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc"} Oct 30 22:23:54.877378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.876991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn" event={"ID":"2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7","Type":"ContainerDied","Data":"58df1eeaab43a677ead8ed6465e6f4c3a44f3e7afebd2f422597a80cc64d64a7"} Oct 30 22:23:54.877378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.877014 5036 scope.go:122] "RemoveContainer" containerID="0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc" Oct 30 22:23:54.888848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.888831 5036 scope.go:122] "RemoveContainer" containerID="0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc" Oct 30 22:23:54.889142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:54.889119 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc\": container with ID starting with 0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc not found: ID does not exist" containerID="0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc" Oct 30 22:23:54.889198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.889148 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc"} err="failed to get container status \"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc\": rpc error: code = NotFound desc = could not find container \"0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc\": container with ID starting with 0ed65bee1aa11e639019782bfd15a7d8ce9ddd86d71145509fd7609a7b5fffdc not found: ID does not exist" Oct 30 22:23:54.903796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.903773 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:54.910243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.910221 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-b99ff7rcn"] Oct 30 22:23:54.953370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:54.953329 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" podUID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" containerName="payload-processing" containerID="cri-o://3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81" gracePeriod=25 Oct 30 22:23:55.099928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.099902 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:55.237451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.237407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\") pod \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " Oct 30 22:23:55.238668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.238643 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\") pod \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " Oct 30 22:23:55.238812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.238717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\") pod \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " Oct 30 22:23:55.238812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.238751 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\") pod \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\" (UID: \"839f5ad2-8a45-45ae-a7e8-65ac3751edf3\") " Oct 30 22:23:55.239655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.239610 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume" pod "839f5ad2-8a45-45ae-a7e8-65ac3751edf3" (UID: "839f5ad2-8a45-45ae-a7e8-65ac3751edf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:55.240226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.240065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume" pod "839f5ad2-8a45-45ae-a7e8-65ac3751edf3" (UID: "839f5ad2-8a45-45ae-a7e8-65ac3751edf3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:55.252322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.249538 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials" pod "839f5ad2-8a45-45ae-a7e8-65ac3751edf3" (UID: "839f5ad2-8a45-45ae-a7e8-65ac3751edf3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:55.254769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.254713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96" pod "839f5ad2-8a45-45ae-a7e8-65ac3751edf3" (UID: "839f5ad2-8a45-45ae-a7e8-65ac3751edf3"). InnerVolumeSpecName "kube-api-access-tvt96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:55.278639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.278617 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:23:55.342121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.342043 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.342544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.342518 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.342674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.342551 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.342674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.342569 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/839f5ad2-8a45-45ae-a7e8-65ac3751edf3-kube-api-access-tvt96\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.443868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.443835 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\") pod \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " Oct 30 22:23:55.444062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.444005 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\") pod \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " Oct 30 22:23:55.444159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.444082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\") pod \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " Oct 30 22:23:55.444159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.444147 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\") pod \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\" (UID: \"2ff96c79-dd9b-434a-8ad2-80880f8c2530\") " Oct 30 22:23:55.444492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.444466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume" pod "2ff96c79-dd9b-434a-8ad2-80880f8c2530" (UID: "2ff96c79-dd9b-434a-8ad2-80880f8c2530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:55.444897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.444866 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume" pod "2ff96c79-dd9b-434a-8ad2-80880f8c2530" (UID: "2ff96c79-dd9b-434a-8ad2-80880f8c2530"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:55.445208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.445182 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.445428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.445411 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff96c79-dd9b-434a-8ad2-80880f8c2530-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.448647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.447383 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj" pod "2ff96c79-dd9b-434a-8ad2-80880f8c2530" (UID: "2ff96c79-dd9b-434a-8ad2-80880f8c2530"). InnerVolumeSpecName "kube-api-access-q7bmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:55.448647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.447867 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials" pod "2ff96c79-dd9b-434a-8ad2-80880f8c2530" (UID: "2ff96c79-dd9b-434a-8ad2-80880f8c2530"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:55.547131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.547088 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.547131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.547118 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7bmj\" (UniqueName: \"kubernetes.io/projected/2ff96c79-dd9b-434a-8ad2-80880f8c2530-kube-api-access-q7bmj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.897224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.897155 5036 generic.go:356] "Generic (PLEG): container finished" podID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" containerID="3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81" exitCode=0 Oct 30 22:23:55.897681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.897316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" event={"ID":"2ff96c79-dd9b-434a-8ad2-80880f8c2530","Type":"ContainerDied","Data":"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81"} Oct 30 22:23:55.897681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.897378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" event={"ID":"2ff96c79-dd9b-434a-8ad2-80880f8c2530","Type":"ContainerDied","Data":"d46ca8113aa960ea9b61e1e1c670145dbcc42507a73db34c24d29c32e6456e9d"} Oct 30 22:23:55.897681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.897414 5036 scope.go:122] "RemoveContainer" containerID="3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81" Oct 30 22:23:55.897681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.897560 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5685d65658-dqm8t" Oct 30 22:23:55.898401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.898375 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:23:55.902158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.902121 5036 generic.go:356] "Generic (PLEG): container finished" podID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" containerID="43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941" exitCode=0 Oct 30 22:23:55.902302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.902216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" event={"ID":"839f5ad2-8a45-45ae-a7e8-65ac3751edf3","Type":"ContainerDied","Data":"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941"} Oct 30 22:23:55.902302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.902280 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" event={"ID":"839f5ad2-8a45-45ae-a7e8-65ac3751edf3","Type":"ContainerDied","Data":"843a66e381590956a1587597396a8232cf1d60c6c0d509e6e6137aa38484aaf6"} Oct 30 22:23:55.902408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.902368 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz" Oct 30 22:23:55.918547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.909893 5036 generic.go:356] "Generic (PLEG): container finished" podID="f24c126b-e324-4a50-92e4-318799ce2870" containerID="e8140be39060ee801ea85ad496afd715865e5485efc02e81849181e889c828e7" exitCode=0 Oct 30 22:23:55.918547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.910200 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" event={"ID":"f24c126b-e324-4a50-92e4-318799ce2870","Type":"ContainerDied","Data":"e8140be39060ee801ea85ad496afd715865e5485efc02e81849181e889c828e7"} Oct 30 22:23:55.922711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.922680 5036 generic.go:356] "Generic (PLEG): container finished" podID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerID="22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04" exitCode=0 Oct 30 22:23:55.922843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.922726 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" Oct 30 22:23:55.922843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.922734 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" event={"ID":"e9ba631b-dc79-4aa7-97c9-9638d3037991","Type":"ContainerDied","Data":"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04"} Oct 30 22:23:55.922843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.922775 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv" event={"ID":"e9ba631b-dc79-4aa7-97c9-9638d3037991","Type":"ContainerDied","Data":"1241eef8210801b79610cd4da3483fbdc9ab88230070613d74146a424754bd31"} Oct 30 22:23:55.925484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.925462 5036 scope.go:122] "RemoveContainer" containerID="3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81" Oct 30 22:23:55.925993 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:55.925849 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81\": container with ID starting with 3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81 not found: ID does not exist" containerID="3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81" Oct 30 22:23:55.925993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.925885 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81"} err="failed to get container status \"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81\": rpc error: code = NotFound desc = could not find container \"3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81\": container with ID starting with 3e1e9646a4bd312af38222009b030665f65dbbd49f16ed5d0639dccb0143ef81 not found: ID does not exist" Oct 30 22:23:55.925993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.925905 5036 scope.go:122] "RemoveContainer" containerID="43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941" Oct 30 22:23:55.982001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.981987 5036 scope.go:122] "RemoveContainer" containerID="43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941" Oct 30 22:23:55.982230 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:55.982211 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941\": container with ID starting with 43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941 not found: ID does not exist" containerID="43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941" Oct 30 22:23:55.982296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.982241 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941"} err="failed to get container status \"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941\": rpc error: code = NotFound desc = could not find container \"43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941\": container with ID starting with 43363f155e0578f00b4ad7c246c63b8df703e7b71d17d673a29acce068c06941 not found: ID does not exist" Oct 30 22:23:55.982347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.982302 5036 scope.go:122] "RemoveContainer" containerID="22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04" Oct 30 22:23:55.983830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.983787 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:23:55.992660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.992639 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:55.996996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.996971 5036 scope.go:122] "RemoveContainer" containerID="22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04" Oct 30 22:23:55.997386 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:23:55.997356 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04\": container with ID starting with 22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04 not found: ID does not exist" containerID="22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04" Oct 30 22:23:55.997470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.997395 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04"} err="failed to get container status \"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04\": rpc error: code = NotFound desc = could not find container \"22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04\": container with ID starting with 22c3477f9567d8a16914c0e47f614165831cf77a3bc070a92038647fb5220b04 not found: ID does not exist" Oct 30 22:23:55.999375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:55.999356 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5777m42qz"] Oct 30 22:23:56.007773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.007742 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:23:56.013533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.013510 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5685d65658-dqm8t"] Oct 30 22:23:56.071039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071003 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071169 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071225 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071258 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071475 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.071535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071484 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:56.071674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071622 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.072539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071735 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\") pod \"e9ba631b-dc79-4aa7-97c9-9638d3037991\" (UID: \"e9ba631b-dc79-4aa7-97c9-9638d3037991\") " Oct 30 22:23:56.072539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.071993 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.072539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.072223 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.072795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.072671 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.072795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.072696 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.072795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.072712 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ba631b-dc79-4aa7-97c9-9638d3037991-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.073394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.073334 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.073560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.073527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.076302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.076253 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:56.076533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.076503 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "kube-api-access-srcch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:56.076677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.076654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:56.076906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.076886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy" pod "e9ba631b-dc79-4aa7-97c9-9638d3037991" (UID: "e9ba631b-dc79-4aa7-97c9-9638d3037991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.173478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173393 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173462 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173554 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173599 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173660 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173713 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173900 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.173941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.173916 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.174054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174006 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.174054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174045 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\") pod \"f24c126b-e324-4a50-92e4-318799ce2870\" (UID: \"f24c126b-e324-4a50-92e4-318799ce2870\") " Oct 30 22:23:56.174522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174251 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.174522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174425 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.174522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174463 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:56.174729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174546 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.174883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174868 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174887 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174898 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srcch\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-kube-api-access-srcch\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174907 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174916 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174924 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.174948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174936 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.175254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174950 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.175254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174961 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24c126b-e324-4a50-92e4-318799ce2870-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.175254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174970 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ba631b-dc79-4aa7-97c9-9638d3037991-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.175254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.174979 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.176424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.176398 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:56.176543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.176404 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:56.176543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.176519 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "kube-api-access-p2c9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:56.176827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.176798 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token" pod "f24c126b-e324-4a50-92e4-318799ce2870" (UID: "f24c126b-e324-4a50-92e4-318799ce2870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:56.190787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.190765 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" path="/var/lib/kubelet/pods/2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7/volumes" Oct 30 22:23:56.191232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.191216 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" path="/var/lib/kubelet/pods/2ff96c79-dd9b-434a-8ad2-80880f8c2530/volumes" Oct 30 22:23:56.191648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.191634 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" path="/var/lib/kubelet/pods/839f5ad2-8a45-45ae-a7e8-65ac3751edf3/volumes" Oct 30 22:23:56.243915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.243876 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:23:56.248653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.248627 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-7hkgv"] Oct 30 22:23:56.276976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.276953 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2c9d\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-kube-api-access-p2c9d\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.277074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.276980 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24c126b-e324-4a50-92e4-318799ce2870-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.277074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.276995 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24c126b-e324-4a50-92e4-318799ce2870-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.277074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.277004 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24c126b-e324-4a50-92e4-318799ce2870-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.988052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.988010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" event={"ID":"f24c126b-e324-4a50-92e4-318799ce2870","Type":"ContainerDied","Data":"1d0ebcde7d2f7b53fefa19325ac1be9c2f4877f529774c58db4d6d4fc0ecae32"} Oct 30 22:23:56.988714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.988090 5036 scope.go:122] "RemoveContainer" containerID="e8140be39060ee801ea85ad496afd715865e5485efc02e81849181e889c828e7" Oct 30 22:23:56.988714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:56.988237 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9" Oct 30 22:23:57.020925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:57.020894 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:23:57.028766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:57.028739 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-swlf9"] Oct 30 22:23:58.332286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:58.328574 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" path="/var/lib/kubelet/pods/e9ba631b-dc79-4aa7-97c9-9638d3037991/volumes" Oct 30 22:23:58.332286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:58.329363 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24c126b-e324-4a50-92e4-318799ce2870" path="/var/lib/kubelet/pods/f24c126b-e324-4a50-92e4-318799ce2870/volumes" Oct 30 22:23:59.782310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.779248 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj"] Oct 30 22:23:59.790329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.790298 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" Oct 30 22:23:59.790548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.790531 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" Oct 30 22:23:59.790706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.790692 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" containerName="payload-processing" Oct 30 22:23:59.790812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.790799 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" containerName="payload-processing" Oct 30 22:23:59.790921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.790910 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" containerName="payload-processing" Oct 30 22:23:59.791011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791000 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" containerName="payload-processing" Oct 30 22:23:59.791120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791109 5036 cpu_manager.go:414] "containerMap: removing container" podUID="67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" containerName="revoke-keys" Oct 30 22:23:59.791213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791200 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" containerName="revoke-keys" Oct 30 22:23:59.791336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791323 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" Oct 30 22:23:59.791434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791423 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" Oct 30 22:23:59.791531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791520 5036 cpu_manager.go:414] "containerMap: removing container" podUID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" containerName="payload-processing" Oct 30 22:23:59.791617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791606 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" containerName="payload-processing" Oct 30 22:23:59.791972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.791953 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ba631b-dc79-4aa7-97c9-9638d3037991" containerName="istio-proxy" Oct 30 22:23:59.792097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.792085 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1c62f5-aeef-4c2b-a347-4c0a1dc59cf7" containerName="payload-processing" Oct 30 22:23:59.792200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.792189 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff96c79-dd9b-434a-8ad2-80880f8c2530" containerName="payload-processing" Oct 30 22:23:59.795368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.795349 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" containerName="revoke-keys" Oct 30 22:23:59.795508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.795494 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24c126b-e324-4a50-92e4-318799ce2870" containerName="istio-proxy" Oct 30 22:23:59.795608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.795597 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="839f5ad2-8a45-45ae-a7e8-65ac3751edf3" containerName="payload-processing" Oct 30 22:23:59.816290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.803702 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:23:59.869321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.865467 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:59.869321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.865696 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:23:59.869321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.865772 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:59.869321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.865823 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:00.016709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.936342 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.016709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.936528 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87v9f\" (UniqueName: \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.016709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:23:59.936632 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.064347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.043427 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.064347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.043619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87v9f\" (UniqueName: \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.064347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.043730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.064347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.044826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.140530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.120795 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj"] Oct 30 22:24:00.262305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.257696 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.282348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.277986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87v9f\" (UniqueName: \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\") pod \"maas-api-revoke-sub-15c8d104-qphpj\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:00.573040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:00.571522 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:01.547155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:01.547122 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj"] Oct 30 22:24:01.817288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:01.816613 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:02.460338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.458680 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" event={"ID":"4bd46718-de10-45f2-b585-22e048672549","Type":"ContainerStarted","Data":"498612f9f7f4846b959b8eb2aae823af08aa4c66abfe99050e7b5d1c256d1d19"} Oct 30 22:24:02.460338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.458783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" event={"ID":"4bd46718-de10-45f2-b585-22e048672549","Type":"ContainerStarted","Data":"b00aa97de031fcc6a15a247d61e8ae9d49274a35d06c6fe7d0edaf1986908456"} Oct 30 22:24:02.529772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.529666 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" podStartSLOduration=3.529647514 podStartE2EDuration="3.529647514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:02.479737812 +0000 UTC m=+2666.429692770" watchObservedRunningTime="2026-10-30 22:24:02.529647514 +0000 UTC m=+2666.479602461" Oct 30 22:24:02.938648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.937908 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:02.946126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.945804 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:02.949936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.949906 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9d868\"/\"agc-cred-f86145528a13\"" Oct 30 22:24:02.953536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:02.953507 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:03.039697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.039659 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7j46\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.039912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.039882 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.040047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.040029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.040122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.040092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.142538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.142494 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.142767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.142668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.142767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.142736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.142959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.142936 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7j46\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.144545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.144395 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.145378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.145341 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.148955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.148931 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.157956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.157938 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7j46\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.291025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.290936 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:03.296457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.296422 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.300605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.300578 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5f192\"/\"agc-cred-6511a467f6d0\"" Oct 30 22:24:03.307377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.307345 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:03.310036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.310012 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:03.347910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.347870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.348071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.348011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.348709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.348135 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.348709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.348507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxh4r\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.450448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.450402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxh4r\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.450668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.450492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.450668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.450540 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.450668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.450615 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.451522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.451489 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.452132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.452096 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.455309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.454899 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.463974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.463944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxh4r\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.473614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.473586 5036 generic.go:356] "Generic (PLEG): container finished" podID="4bd46718-de10-45f2-b585-22e048672549" containerID="498612f9f7f4846b959b8eb2aae823af08aa4c66abfe99050e7b5d1c256d1d19" exitCode=0 Oct 30 22:24:03.473748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.473728 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" event={"ID":"4bd46718-de10-45f2-b585-22e048672549","Type":"ContainerDied","Data":"498612f9f7f4846b959b8eb2aae823af08aa4c66abfe99050e7b5d1c256d1d19"} Oct 30 22:24:03.617187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.617102 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:03.638185 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:03.638149 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2305c474_7344_4c9d_b54a_9c57ccfcc449.slice/crio-9e6c63f021947c9a7b0b579fc04bc4478ba7ca3f91e813266b45f0626a0b70b1 WatchSource:0}: Error finding container 9e6c63f021947c9a7b0b579fc04bc4478ba7ca3f91e813266b45f0626a0b70b1: Status 404 returned error can't find the container with id 9e6c63f021947c9a7b0b579fc04bc4478ba7ca3f91e813266b45f0626a0b70b1 Oct 30 22:24:03.647145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.646666 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:03.931721 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:03.931686 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb0fde3_d5f2_4052_9cb4_044fc92153d0.slice/crio-19de84ca46d87b89ad4fb97ebc52bab9bca56f1ef200a1e9c5bf47bc26229f70 WatchSource:0}: Error finding container 19de84ca46d87b89ad4fb97ebc52bab9bca56f1ef200a1e9c5bf47bc26229f70: Status 404 returned error can't find the container with id 19de84ca46d87b89ad4fb97ebc52bab9bca56f1ef200a1e9c5bf47bc26229f70 Oct 30 22:24:03.933316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:03.933283 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:04.484558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.484516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" event={"ID":"6bb0fde3-d5f2-4052-9cb4-044fc92153d0","Type":"ContainerStarted","Data":"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94"} Oct 30 22:24:04.485009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.484582 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:04.485009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.484604 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" event={"ID":"6bb0fde3-d5f2-4052-9cb4-044fc92153d0","Type":"ContainerStarted","Data":"19de84ca46d87b89ad4fb97ebc52bab9bca56f1ef200a1e9c5bf47bc26229f70"} Oct 30 22:24:04.486455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.486428 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" event={"ID":"2305c474-7344-4c9d-b54a-9c57ccfcc449","Type":"ContainerStarted","Data":"0cdae122270d20b0ee072e3799f600d64abad217adee2da62a7a6e2042293667"} Oct 30 22:24:04.486577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.486479 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" event={"ID":"2305c474-7344-4c9d-b54a-9c57ccfcc449","Type":"ContainerStarted","Data":"9e6c63f021947c9a7b0b579fc04bc4478ba7ca3f91e813266b45f0626a0b70b1"} Oct 30 22:24:04.486639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.486621 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:04.506245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.506195 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" podStartSLOduration=1.506176806 podStartE2EDuration="1.506176806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:04.503017628 +0000 UTC m=+2668.452972635" watchObservedRunningTime="2026-10-30 22:24:04.506176806 +0000 UTC m=+2668.456131753" Oct 30 22:24:04.534554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.534493 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" podStartSLOduration=2.534473545 podStartE2EDuration="2.534473545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:04.53180503 +0000 UTC m=+2668.481759979" watchObservedRunningTime="2026-10-30 22:24:04.534473545 +0000 UTC m=+2668.484428493" Oct 30 22:24:04.759695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.759624 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:04.880460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.880419 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:24:04.880641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.880524 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:24:04.881448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.881426 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\" (UniqueName: \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\") pod \"4bd46718-de10-45f2-b585-22e048672549\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " Oct 30 22:24:04.881585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.881569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\") pod \"4bd46718-de10-45f2-b585-22e048672549\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " Oct 30 22:24:04.881657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.881630 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\") pod \"4bd46718-de10-45f2-b585-22e048672549\" (UID: \"4bd46718-de10-45f2-b585-22e048672549\") " Oct 30 22:24:04.881980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.881959 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca" pod "4bd46718-de10-45f2-b585-22e048672549" (UID: "4bd46718-de10-45f2-b585-22e048672549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:04.882336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.882318 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd46718-de10-45f2-b585-22e048672549-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:04.883830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.883805 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f" pod "4bd46718-de10-45f2-b585-22e048672549" (UID: "4bd46718-de10-45f2-b585-22e048672549"). InnerVolumeSpecName "kube-api-access-87v9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:04.883969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.883950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca" pod "4bd46718-de10-45f2-b585-22e048672549" (UID: "4bd46718-de10-45f2-b585-22e048672549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:04.984937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.984901 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd46718-de10-45f2-b585-22e048672549-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:04.984937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:04.984936 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87v9f\" (UniqueName: \"kubernetes.io/projected/4bd46718-de10-45f2-b585-22e048672549-kube-api-access-87v9f\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:05.497652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:05.497622 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" Oct 30 22:24:05.497652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:05.497644 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj" event={"ID":"4bd46718-de10-45f2-b585-22e048672549","Type":"ContainerDied","Data":"b00aa97de031fcc6a15a247d61e8ae9d49274a35d06c6fe7d0edaf1986908456"} Oct 30 22:24:05.498102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:05.497693 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00aa97de031fcc6a15a247d61e8ae9d49274a35d06c6fe7d0edaf1986908456" Oct 30 22:24:09.461991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:09.461952 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:09.464975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:09.464950 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:10.040720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.040689 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh"] Oct 30 22:24:10.041587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.041570 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4bd46718-de10-45f2-b585-22e048672549" containerName="revoke-keys" Oct 30 22:24:10.041653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.041587 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd46718-de10-45f2-b585-22e048672549" containerName="revoke-keys" Oct 30 22:24:10.041732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.041723 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd46718-de10-45f2-b585-22e048672549" containerName="revoke-keys" Oct 30 22:24:10.045878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.045860 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.050757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.050731 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:10.050884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.050868 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:24:10.050958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.050901 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:10.056430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.053936 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:10.056430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.055481 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh"] Oct 30 22:24:10.118407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.118381 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:10.121580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.121557 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:10.121719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.121553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbj9w\" (UniqueName: \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.121719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.121634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.121903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.121877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.223109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.223064 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.223501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.223474 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj9w\" (UniqueName: \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.223589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.223541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.225316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.223820 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.227529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.227502 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.265303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.263179 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj9w\" (UniqueName: \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\") pod \"maas-api-revoke-sub-f2b878d9-9nggh\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.383902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.382523 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:10.852318 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:10.852281 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686b449_62f1_4d6c_8bba_9388c7cab3e2.slice/crio-1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850 WatchSource:0}: Error finding container 1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850: Status 404 returned error can't find the container with id 1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850 Oct 30 22:24:10.854552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:10.854522 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh"] Oct 30 22:24:11.359047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.359013 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:11.384734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.384702 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.392051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.392015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:11.591859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.591820 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtfv\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.592054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.592030 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.592106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.592084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.592165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.592150 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.604702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.604668 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" event={"ID":"0686b449-62f1-4d6c-8bba-9388c7cab3e2","Type":"ContainerStarted","Data":"9c6b1fb6649c2cc6d5c25985f57093c5aea05c04d984536b75e837698faafd96"} Oct 30 22:24:11.604885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.604733 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" event={"ID":"0686b449-62f1-4d6c-8bba-9388c7cab3e2","Type":"ContainerStarted","Data":"1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850"} Oct 30 22:24:11.625253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.625196 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" podStartSLOduration=1.625180876 podStartE2EDuration="1.625180876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:11.623020704 +0000 UTC m=+2675.572975651" watchObservedRunningTime="2026-10-30 22:24:11.625180876 +0000 UTC m=+2675.575135824" Oct 30 22:24:11.691385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.691298 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:11.693619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.693584 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtfv\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.694216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.694183 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.694540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.694247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.694772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.694746 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.695984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.695932 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.696536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.696514 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.698439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.698420 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.698552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.698417 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.701926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.701624 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2"] Oct 30 22:24:11.753304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.748069 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtfv\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\") pod \"payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:11.753304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.751822 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.804340 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805233 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805360 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805399 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805424 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805468 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj6g\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805698 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46rn\" (UniqueName: \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.816299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.805785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.892321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.858179 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2"] Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907635 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907660 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907694 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj6g\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907861 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46rn\" (UniqueName: \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.907943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.908045 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.909673 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.910294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.914018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.911288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:11.998420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.998325 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.998847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.998821 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46rn\" (UniqueName: \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\") pod \"maas-api-revoke-sub-843f4177-cbld2\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:11.999175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:11.999153 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:12.002068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.002020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj6g\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:12.007649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.007575 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:12.151257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.150065 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:12.197838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.197808 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:12.385376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.385230 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:12.491154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.491122 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:12.618008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.617980 5036 generic.go:356] "Generic (PLEG): container finished" podID="0686b449-62f1-4d6c-8bba-9388c7cab3e2" containerID="9c6b1fb6649c2cc6d5c25985f57093c5aea05c04d984536b75e837698faafd96" exitCode=0 Oct 30 22:24:12.618168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.618070 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" event={"ID":"0686b449-62f1-4d6c-8bba-9388c7cab3e2","Type":"ContainerDied","Data":"9c6b1fb6649c2cc6d5c25985f57093c5aea05c04d984536b75e837698faafd96"} Oct 30 22:24:12.620235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.620210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" event={"ID":"12985313-8ab2-42c6-be83-565725236808","Type":"ContainerStarted","Data":"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75"} Oct 30 22:24:12.620370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.620259 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" event={"ID":"12985313-8ab2-42c6-be83-565725236808","Type":"ContainerStarted","Data":"2dafc19dc608c09b8293a34c90f0b749751c54426904a1c59ead04b41c9e6830"} Oct 30 22:24:12.620370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.620306 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:12.622646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.622624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" event={"ID":"014e830d-a7a0-4caa-862f-a76bec6f5bc3","Type":"ContainerStarted","Data":"60acfa59a77fe24d6ac64efa3f368fb9c28f5cd1ee255008091ac9237420b43f"} Oct 30 22:24:12.622772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.622672 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" event={"ID":"014e830d-a7a0-4caa-862f-a76bec6f5bc3","Type":"ContainerStarted","Data":"1a6a710e9802ff0c2457534e81c5fa2f5a98388a8e961780cdf86aab8c7cbcc7"} Oct 30 22:24:12.622772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.622703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:12.639803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.639769 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2"] Oct 30 22:24:12.640215 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:12.640185 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7a22f8_f993_43bb_bbcc_1671af83a63e.slice/crio-d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780 WatchSource:0}: Error finding container d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780: Status 404 returned error can't find the container with id d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780 Oct 30 22:24:12.659572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.659528 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" podStartSLOduration=1.659511473 podStartE2EDuration="1.659511473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:12.65694224 +0000 UTC m=+2676.606897187" watchObservedRunningTime="2026-10-30 22:24:12.659511473 +0000 UTC m=+2676.609466418" Oct 30 22:24:12.677120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:12.677071 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" podStartSLOduration=1.677055288 podStartE2EDuration="1.677055288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:12.676149727 +0000 UTC m=+2676.626104672" watchObservedRunningTime="2026-10-30 22:24:12.677055288 +0000 UTC m=+2676.627010235" Oct 30 22:24:13.645254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.645219 5036 generic.go:356] "Generic (PLEG): container finished" podID="dc7a22f8-f993-43bb-bbcc-1671af83a63e" containerID="bde7ce85755edbe733dee513895686a8cfdee1ab016a0349045ba10fa90061f2" exitCode=0 Oct 30 22:24:13.645672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.645307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" event={"ID":"dc7a22f8-f993-43bb-bbcc-1671af83a63e","Type":"ContainerDied","Data":"bde7ce85755edbe733dee513895686a8cfdee1ab016a0349045ba10fa90061f2"} Oct 30 22:24:13.645672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.645373 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" event={"ID":"dc7a22f8-f993-43bb-bbcc-1671af83a63e","Type":"ContainerStarted","Data":"d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780"} Oct 30 22:24:13.838524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.838494 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:13.954550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.954517 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\") pod \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " Oct 30 22:24:13.954764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.954584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\") pod \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " Oct 30 22:24:13.954764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.954750 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\" (UniqueName: \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\") pod \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\" (UID: \"0686b449-62f1-4d6c-8bba-9388c7cab3e2\") " Oct 30 22:24:13.955024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.954992 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca" pod "0686b449-62f1-4d6c-8bba-9388c7cab3e2" (UID: "0686b449-62f1-4d6c-8bba-9388c7cab3e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:13.955492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.955472 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:13.957007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.956977 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca" pod "0686b449-62f1-4d6c-8bba-9388c7cab3e2" (UID: "0686b449-62f1-4d6c-8bba-9388c7cab3e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:13.957507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:13.957483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w" pod "0686b449-62f1-4d6c-8bba-9388c7cab3e2" (UID: "0686b449-62f1-4d6c-8bba-9388c7cab3e2"). InnerVolumeSpecName "kube-api-access-tbj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:14.057125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.057039 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0686b449-62f1-4d6c-8bba-9388c7cab3e2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.057125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.057076 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbj9w\" (UniqueName: \"kubernetes.io/projected/0686b449-62f1-4d6c-8bba-9388c7cab3e2-kube-api-access-tbj9w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.462727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.462666 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" podUID="2305c474-7344-4c9d-b54a-9c57ccfcc449" containerName="payload-processing" containerID="cri-o://0cdae122270d20b0ee072e3799f600d64abad217adee2da62a7a6e2042293667" gracePeriod=25 Oct 30 22:24:14.675060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.675021 5036 generic.go:356] "Generic (PLEG): container finished" podID="2305c474-7344-4c9d-b54a-9c57ccfcc449" containerID="0cdae122270d20b0ee072e3799f600d64abad217adee2da62a7a6e2042293667" exitCode=0 Oct 30 22:24:14.675540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.675179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" event={"ID":"2305c474-7344-4c9d-b54a-9c57ccfcc449","Type":"ContainerDied","Data":"0cdae122270d20b0ee072e3799f600d64abad217adee2da62a7a6e2042293667"} Oct 30 22:24:14.677204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.677183 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" Oct 30 22:24:14.678113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.677351 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh" event={"ID":"0686b449-62f1-4d6c-8bba-9388c7cab3e2","Type":"ContainerDied","Data":"1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850"} Oct 30 22:24:14.678113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.677412 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee33d871fe97afcdb4264d4e41eecdf3b36c01ee6d6797496c1c0ff0fff2850" Oct 30 22:24:14.784689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.784664 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:14.872500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.872468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\") pod \"2305c474-7344-4c9d-b54a-9c57ccfcc449\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " Oct 30 22:24:14.872741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.872725 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\") pod \"2305c474-7344-4c9d-b54a-9c57ccfcc449\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " Oct 30 22:24:14.872909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.872776 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\") pod \"2305c474-7344-4c9d-b54a-9c57ccfcc449\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " Oct 30 22:24:14.872909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.872887 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\") pod \"2305c474-7344-4c9d-b54a-9c57ccfcc449\" (UID: \"2305c474-7344-4c9d-b54a-9c57ccfcc449\") " Oct 30 22:24:14.873234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.873145 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume" pod "2305c474-7344-4c9d-b54a-9c57ccfcc449" (UID: "2305c474-7344-4c9d-b54a-9c57ccfcc449"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:14.873359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.873329 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume" pod "2305c474-7344-4c9d-b54a-9c57ccfcc449" (UID: "2305c474-7344-4c9d-b54a-9c57ccfcc449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:14.873888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.873866 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.873957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.873897 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2305c474-7344-4c9d-b54a-9c57ccfcc449-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.875542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.875520 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials" pod "2305c474-7344-4c9d-b54a-9c57ccfcc449" (UID: "2305c474-7344-4c9d-b54a-9c57ccfcc449"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:14.875732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.875714 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46" pod "2305c474-7344-4c9d-b54a-9c57ccfcc449" (UID: "2305c474-7344-4c9d-b54a-9c57ccfcc449"). InnerVolumeSpecName "kube-api-access-j7j46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:14.904216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.904194 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.984542 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\" (UniqueName: \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\") pod \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.985048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\") pod \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.985681 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\") pod \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\" (UID: \"dc7a22f8-f993-43bb-bbcc-1671af83a63e\") " Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.986189 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca" pod "dc7a22f8-f993-43bb-bbcc-1671af83a63e" (UID: "dc7a22f8-f993-43bb-bbcc-1671af83a63e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.987184 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7j46\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-kube-api-access-j7j46\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.987217 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2305c474-7344-4c9d-b54a-9c57ccfcc449-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.989340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.987232 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:14.993413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.991949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca" pod "dc7a22f8-f993-43bb-bbcc-1671af83a63e" (UID: "dc7a22f8-f993-43bb-bbcc-1671af83a63e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:14.993413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:14.992583 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn" pod "dc7a22f8-f993-43bb-bbcc-1671af83a63e" (UID: "dc7a22f8-f993-43bb-bbcc-1671af83a63e"). InnerVolumeSpecName "kube-api-access-s46rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:15.089205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.089173 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7a22f8-f993-43bb-bbcc-1671af83a63e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.089205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.089203 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s46rn\" (UniqueName: \"kubernetes.io/projected/dc7a22f8-f993-43bb-bbcc-1671af83a63e-kube-api-access-s46rn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.118992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.118949 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" podUID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" containerName="payload-processing" containerID="cri-o://33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94" gracePeriod=25 Oct 30 22:24:15.441167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.441142 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:15.495884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.495853 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\") pod \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " Oct 30 22:24:15.496161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496146 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\") pod \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " Oct 30 22:24:15.496239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\") pod \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " Oct 30 22:24:15.496359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496290 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\") pod \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\" (UID: \"6bb0fde3-d5f2-4052-9cb4-044fc92153d0\") " Oct 30 22:24:15.496490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496286 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume" pod "6bb0fde3-d5f2-4052-9cb4-044fc92153d0" (UID: "6bb0fde3-d5f2-4052-9cb4-044fc92153d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:15.496677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume" pod "6bb0fde3-d5f2-4052-9cb4-044fc92153d0" (UID: "6bb0fde3-d5f2-4052-9cb4-044fc92153d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:15.497011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.496995 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.497074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.497015 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.498697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.498667 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r" pod "6bb0fde3-d5f2-4052-9cb4-044fc92153d0" (UID: "6bb0fde3-d5f2-4052-9cb4-044fc92153d0"). InnerVolumeSpecName "kube-api-access-mxh4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:15.499217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.499200 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials" pod "6bb0fde3-d5f2-4052-9cb4-044fc92153d0" (UID: "6bb0fde3-d5f2-4052-9cb4-044fc92153d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:15.599648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.599546 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.599648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.599588 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxh4r\" (UniqueName: \"kubernetes.io/projected/6bb0fde3-d5f2-4052-9cb4-044fc92153d0-kube-api-access-mxh4r\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:15.687958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.687925 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" Oct 30 22:24:15.688475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.687962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7" event={"ID":"2305c474-7344-4c9d-b54a-9c57ccfcc449","Type":"ContainerDied","Data":"9e6c63f021947c9a7b0b579fc04bc4478ba7ca3f91e813266b45f0626a0b70b1"} Oct 30 22:24:15.688475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.688031 5036 scope.go:122] "RemoveContainer" containerID="0cdae122270d20b0ee072e3799f600d64abad217adee2da62a7a6e2042293667" Oct 30 22:24:15.689996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.689972 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" Oct 30 22:24:15.690114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.690003 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2" event={"ID":"dc7a22f8-f993-43bb-bbcc-1671af83a63e","Type":"ContainerDied","Data":"d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780"} Oct 30 22:24:15.690114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.690050 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b27c4642c9fa0c13a12545e57cca1f71012ef58aba2b0e363c2779e6b5b780" Oct 30 22:24:15.692352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.692322 5036 generic.go:356] "Generic (PLEG): container finished" podID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" containerID="33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94" exitCode=0 Oct 30 22:24:15.692492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.692398 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" Oct 30 22:24:15.692492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.692430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" event={"ID":"6bb0fde3-d5f2-4052-9cb4-044fc92153d0","Type":"ContainerDied","Data":"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94"} Oct 30 22:24:15.692492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.692476 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn" event={"ID":"6bb0fde3-d5f2-4052-9cb4-044fc92153d0","Type":"ContainerDied","Data":"19de84ca46d87b89ad4fb97ebc52bab9bca56f1ef200a1e9c5bf47bc26229f70"} Oct 30 22:24:15.780819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.780789 5036 scope.go:122] "RemoveContainer" containerID="33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94" Oct 30 22:24:15.781765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.781737 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:15.787361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.787337 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-75dbqtjp7"] Oct 30 22:24:15.792217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.792193 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:15.802301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.800065 5036 scope.go:122] "RemoveContainer" containerID="33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94" Oct 30 22:24:15.802301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.800237 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5bbb5x5cn"] Oct 30 22:24:15.802301 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:15.800492 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94\": container with ID starting with 33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94 not found: ID does not exist" containerID="33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94" Oct 30 22:24:15.802301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:15.800527 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94"} err="failed to get container status \"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94\": rpc error: code = NotFound desc = could not find container \"33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94\": container with ID starting with 33acac59e5d81151bf443c87648d55ebd62ace703685d09d5e492290abce3f94 not found: ID does not exist" Oct 30 22:24:16.190983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:16.190950 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2305c474-7344-4c9d-b54a-9c57ccfcc449" path="/var/lib/kubelet/pods/2305c474-7344-4c9d-b54a-9c57ccfcc449/volumes" Oct 30 22:24:16.191486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:16.191471 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" path="/var/lib/kubelet/pods/6bb0fde3-d5f2-4052-9cb4-044fc92153d0/volumes" Oct 30 22:24:18.104828 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:18.104780 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:24:20.539021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.538988 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528"] Oct 30 22:24:20.539843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539826 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" containerName="payload-processing" Oct 30 22:24:20.539896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539845 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" containerName="payload-processing" Oct 30 22:24:20.539896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539865 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0686b449-62f1-4d6c-8bba-9388c7cab3e2" containerName="revoke-keys" Oct 30 22:24:20.539896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539870 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686b449-62f1-4d6c-8bba-9388c7cab3e2" containerName="revoke-keys" Oct 30 22:24:20.539896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539894 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2305c474-7344-4c9d-b54a-9c57ccfcc449" containerName="payload-processing" Oct 30 22:24:20.540044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539900 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2305c474-7344-4c9d-b54a-9c57ccfcc449" containerName="payload-processing" Oct 30 22:24:20.540044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539914 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dc7a22f8-f993-43bb-bbcc-1671af83a63e" containerName="revoke-keys" Oct 30 22:24:20.540044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.539919 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7a22f8-f993-43bb-bbcc-1671af83a63e" containerName="revoke-keys" Oct 30 22:24:20.540044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.540033 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2305c474-7344-4c9d-b54a-9c57ccfcc449" containerName="payload-processing" Oct 30 22:24:20.540044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.540042 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7a22f8-f993-43bb-bbcc-1671af83a63e" containerName="revoke-keys" Oct 30 22:24:20.540199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.540054 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb0fde3-d5f2-4052-9cb4-044fc92153d0" containerName="payload-processing" Oct 30 22:24:20.540199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.540063 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686b449-62f1-4d6c-8bba-9388c7cab3e2" containerName="revoke-keys" Oct 30 22:24:20.545551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.545522 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.559449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.558515 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528"] Oct 30 22:24:20.567049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.567024 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:20.567288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.567025 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:20.567425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.567088 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:20.567570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.567156 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:24:20.600928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.600881 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.601142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.601055 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdxz\" (UniqueName: \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.601214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.601181 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.702624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.702581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.702809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.702728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdxz\" (UniqueName: \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.702907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.702836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.703254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.703232 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.705642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.705618 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.715630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.715606 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdxz\" (UniqueName: \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\") pod \"maas-api-revoke-sub-9efd6f3e-wq528\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:20.867096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:20.867002 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:21.157870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:21.157837 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528"] Oct 30 22:24:21.158723 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:21.158694 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482b83d4_e366_46cb_a75d_6b0522dc607f.slice/crio-af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1 WatchSource:0}: Error finding container af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1: Status 404 returned error can't find the container with id af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1 Oct 30 22:24:21.760703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:21.760658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" event={"ID":"482b83d4-e366-46cb-a75d-6b0522dc607f","Type":"ContainerStarted","Data":"3d40e7563399c4992cda7bb5c23eb730de63062327ec9407f84873f03102e3b8"} Oct 30 22:24:21.761206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:21.760726 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" event={"ID":"482b83d4-e366-46cb-a75d-6b0522dc607f","Type":"ContainerStarted","Data":"af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1"} Oct 30 22:24:21.779862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:21.779758 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" podStartSLOduration=1.779744236 podStartE2EDuration="1.779744236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:21.777121609 +0000 UTC m=+2685.727076556" watchObservedRunningTime="2026-10-30 22:24:21.779744236 +0000 UTC m=+2685.729699181" Oct 30 22:24:22.218400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.218372 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95"] Oct 30 22:24:22.224329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.224300 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.232821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.232793 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95"] Oct 30 22:24:22.352140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.352102 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.352386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.352324 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrl2\" (UniqueName: \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.352527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.352507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.454084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.454003 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.454242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.454103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrl2\" (UniqueName: \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.454242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.454234 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.455022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.454997 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.456885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.456859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.466924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.466897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrl2\" (UniqueName: \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\") pod \"maas-api-revoke-sub-ec5845f6-mnw95\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.556158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.556125 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:22.771813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.771728 5036 generic.go:356] "Generic (PLEG): container finished" podID="482b83d4-e366-46cb-a75d-6b0522dc607f" containerID="3d40e7563399c4992cda7bb5c23eb730de63062327ec9407f84873f03102e3b8" exitCode=0 Oct 30 22:24:22.772203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.771849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" event={"ID":"482b83d4-e366-46cb-a75d-6b0522dc607f","Type":"ContainerDied","Data":"3d40e7563399c4992cda7bb5c23eb730de63062327ec9407f84873f03102e3b8"} Oct 30 22:24:22.850403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:22.850373 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95"] Oct 30 22:24:23.648662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.648629 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:23.648839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.648715 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:23.713319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.710587 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:24:23.734469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.734438 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:24:23.791975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.791871 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" event={"ID":"cd1990fc-e9c9-4ab2-b776-9a097950ce59","Type":"ContainerStarted","Data":"2ea2ae4dbd725f5f294b440ba272b771bf30eca38c78a80fa232b4f5437d8f6f"} Oct 30 22:24:23.791975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.791930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" event={"ID":"cd1990fc-e9c9-4ab2-b776-9a097950ce59","Type":"ContainerStarted","Data":"a5e471d8c3213450ed3fbaa3e8bbbe6af91a3a17606ddeb11cfbc232389cd8e8"} Oct 30 22:24:23.820507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:23.818732 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" podStartSLOduration=1.818715637 podStartE2EDuration="1.818715637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:23.815680534 +0000 UTC m=+2687.765635480" watchObservedRunningTime="2026-10-30 22:24:23.818715637 +0000 UTC m=+2687.768670584" Oct 30 22:24:24.013177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.013146 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:24.197178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.197103 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\") pod \"482b83d4-e366-46cb-a75d-6b0522dc607f\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " Oct 30 22:24:24.197355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.197251 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\") pod \"482b83d4-e366-46cb-a75d-6b0522dc607f\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " Oct 30 22:24:24.197417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.197364 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\" (UniqueName: \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\") pod \"482b83d4-e366-46cb-a75d-6b0522dc607f\" (UID: \"482b83d4-e366-46cb-a75d-6b0522dc607f\") " Oct 30 22:24:24.197603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.197580 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca" pod "482b83d4-e366-46cb-a75d-6b0522dc607f" (UID: "482b83d4-e366-46cb-a75d-6b0522dc607f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:24.197945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.197927 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.199494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.199467 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca" pod "482b83d4-e366-46cb-a75d-6b0522dc607f" (UID: "482b83d4-e366-46cb-a75d-6b0522dc607f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:24.199662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.199641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz" pod "482b83d4-e366-46cb-a75d-6b0522dc607f" (UID: "482b83d4-e366-46cb-a75d-6b0522dc607f"). InnerVolumeSpecName "kube-api-access-qcdxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:24.299815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.299770 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482b83d4-e366-46cb-a75d-6b0522dc607f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.299815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.299803 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdxz\" (UniqueName: \"kubernetes.io/projected/482b83d4-e366-46cb-a75d-6b0522dc607f-kube-api-access-qcdxz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.802001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.801970 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" Oct 30 22:24:24.802488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.802012 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528" event={"ID":"482b83d4-e366-46cb-a75d-6b0522dc607f","Type":"ContainerDied","Data":"af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1"} Oct 30 22:24:24.802488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.802072 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af018bde7ef1f0828c4f204d056f211cabd2818a9c0c8e9f7be2d00d633379c1" Oct 30 22:24:24.804032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.804009 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd1990fc-e9c9-4ab2-b776-9a097950ce59" containerID="2ea2ae4dbd725f5f294b440ba272b771bf30eca38c78a80fa232b4f5437d8f6f" exitCode=0 Oct 30 22:24:24.804146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:24.804105 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" event={"ID":"cd1990fc-e9c9-4ab2-b776-9a097950ce59","Type":"ContainerDied","Data":"2ea2ae4dbd725f5f294b440ba272b771bf30eca38c78a80fa232b4f5437d8f6f"} Oct 30 22:24:26.057588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.057558 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:26.236673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.236410 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\") pod \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " Oct 30 22:24:26.236673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.236504 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\" (UniqueName: \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\") pod \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " Oct 30 22:24:26.236673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.236640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\") pod \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\" (UID: \"cd1990fc-e9c9-4ab2-b776-9a097950ce59\") " Oct 30 22:24:26.237074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.236990 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca" pod "cd1990fc-e9c9-4ab2-b776-9a097950ce59" (UID: "cd1990fc-e9c9-4ab2-b776-9a097950ce59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:26.237806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.237778 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:26.240341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.239418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2" pod "cd1990fc-e9c9-4ab2-b776-9a097950ce59" (UID: "cd1990fc-e9c9-4ab2-b776-9a097950ce59"). InnerVolumeSpecName "kube-api-access-mrrl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:26.240518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.240497 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca" pod "cd1990fc-e9c9-4ab2-b776-9a097950ce59" (UID: "cd1990fc-e9c9-4ab2-b776-9a097950ce59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:26.338907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.338873 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrl2\" (UniqueName: \"kubernetes.io/projected/cd1990fc-e9c9-4ab2-b776-9a097950ce59-kube-api-access-mrrl2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:26.338907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.338900 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1990fc-e9c9-4ab2-b776-9a097950ce59-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:26.828508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.828481 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" Oct 30 22:24:26.828697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.828479 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95" event={"ID":"cd1990fc-e9c9-4ab2-b776-9a097950ce59","Type":"ContainerDied","Data":"a5e471d8c3213450ed3fbaa3e8bbbe6af91a3a17606ddeb11cfbc232389cd8e8"} Oct 30 22:24:26.828697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:26.828616 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e471d8c3213450ed3fbaa3e8bbbe6af91a3a17606ddeb11cfbc232389cd8e8" Oct 30 22:24:28.711596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:28.711557 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" podUID="1c066c74-fcd2-4796-93f5-20640daf4bdc" containerName="payload-processing" containerID="cri-o://5a9ddfea7e675b3029b4e1e89b26ea6e15139aad72bf0d556e3dd2a4a53e40e8" gracePeriod=25 Oct 30 22:24:28.735202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:28.735161 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" podUID="234f8516-68cd-4216-8e00-fced9a946778" containerName="payload-processing" containerID="cri-o://22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea" gracePeriod=25 Oct 30 22:24:28.855825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:28.855631 5036 generic.go:356] "Generic (PLEG): container finished" podID="1c066c74-fcd2-4796-93f5-20640daf4bdc" containerID="5a9ddfea7e675b3029b4e1e89b26ea6e15139aad72bf0d556e3dd2a4a53e40e8" exitCode=0 Oct 30 22:24:28.856159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:28.856128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" event={"ID":"1c066c74-fcd2-4796-93f5-20640daf4bdc","Type":"ContainerDied","Data":"5a9ddfea7e675b3029b4e1e89b26ea6e15139aad72bf0d556e3dd2a4a53e40e8"} Oct 30 22:24:29.128383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.128354 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:24:29.180627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.180598 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:24:29.202109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202077 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\") pod \"234f8516-68cd-4216-8e00-fced9a946778\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " Oct 30 22:24:29.202302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202169 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\") pod \"1c066c74-fcd2-4796-93f5-20640daf4bdc\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " Oct 30 22:24:29.202302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\") pod \"1c066c74-fcd2-4796-93f5-20640daf4bdc\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " Oct 30 22:24:29.202466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\") pod \"234f8516-68cd-4216-8e00-fced9a946778\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " Oct 30 22:24:29.202466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202384 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\") pod \"234f8516-68cd-4216-8e00-fced9a946778\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " Oct 30 22:24:29.202596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202485 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\") pod \"1c066c74-fcd2-4796-93f5-20640daf4bdc\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " Oct 30 22:24:29.202663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202589 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume" pod "234f8516-68cd-4216-8e00-fced9a946778" (UID: "234f8516-68cd-4216-8e00-fced9a946778"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.202732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202665 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\") pod \"1c066c74-fcd2-4796-93f5-20640daf4bdc\" (UID: \"1c066c74-fcd2-4796-93f5-20640daf4bdc\") " Oct 30 22:24:29.202732 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.202704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\") pod \"234f8516-68cd-4216-8e00-fced9a946778\" (UID: \"234f8516-68cd-4216-8e00-fced9a946778\") " Oct 30 22:24:29.203540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.203515 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.203643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.203561 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume" pod "234f8516-68cd-4216-8e00-fced9a946778" (UID: "234f8516-68cd-4216-8e00-fced9a946778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.203767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.203733 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume" pod "1c066c74-fcd2-4796-93f5-20640daf4bdc" (UID: "1c066c74-fcd2-4796-93f5-20640daf4bdc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.204466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.204418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume" pod "1c066c74-fcd2-4796-93f5-20640daf4bdc" (UID: "1c066c74-fcd2-4796-93f5-20640daf4bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.207143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.207114 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials" pod "234f8516-68cd-4216-8e00-fced9a946778" (UID: "234f8516-68cd-4216-8e00-fced9a946778"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.208780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.208736 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp" pod "234f8516-68cd-4216-8e00-fced9a946778" (UID: "234f8516-68cd-4216-8e00-fced9a946778"). InnerVolumeSpecName "kube-api-access-rw4jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.210626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.210598 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials" pod "1c066c74-fcd2-4796-93f5-20640daf4bdc" (UID: "1c066c74-fcd2-4796-93f5-20640daf4bdc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.211482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.211455 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf" pod "1c066c74-fcd2-4796-93f5-20640daf4bdc" (UID: "1c066c74-fcd2-4796-93f5-20640daf4bdc"). InnerVolumeSpecName "kube-api-access-cdbxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305296 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305323 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305333 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305342 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234f8516-68cd-4216-8e00-fced9a946778-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305351 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbxf\" (UniqueName: \"kubernetes.io/projected/1c066c74-fcd2-4796-93f5-20640daf4bdc-kube-api-access-cdbxf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305361 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c066c74-fcd2-4796-93f5-20640daf4bdc-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.305369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.305370 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4jp\" (UniqueName: \"kubernetes.io/projected/234f8516-68cd-4216-8e00-fced9a946778-kube-api-access-rw4jp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.867105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.867068 5036 generic.go:356] "Generic (PLEG): container finished" podID="234f8516-68cd-4216-8e00-fced9a946778" containerID="22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea" exitCode=0 Oct 30 22:24:29.867610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.867129 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" Oct 30 22:24:29.867610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.867154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" event={"ID":"234f8516-68cd-4216-8e00-fced9a946778","Type":"ContainerDied","Data":"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea"} Oct 30 22:24:29.867610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.867244 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx" event={"ID":"234f8516-68cd-4216-8e00-fced9a946778","Type":"ContainerDied","Data":"4574941b9a3830127733b869999e1f8c0acd71599dc68968ed41f5ca05a8e3fa"} Oct 30 22:24:29.867610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.867309 5036 scope.go:122] "RemoveContainer" containerID="22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea" Oct 30 22:24:29.869091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.869067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" event={"ID":"1c066c74-fcd2-4796-93f5-20640daf4bdc","Type":"ContainerDied","Data":"dc1bf874687de3324346df7e9cba651731a3128bbbdb7acb73692fe4eab39ac9"} Oct 30 22:24:29.869245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.869220 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g" Oct 30 22:24:29.882712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.882695 5036 scope.go:122] "RemoveContainer" containerID="22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea" Oct 30 22:24:29.882989 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:29.882965 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea\": container with ID starting with 22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea not found: ID does not exist" containerID="22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea" Oct 30 22:24:29.883053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.882996 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea"} err="failed to get container status \"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea\": rpc error: code = NotFound desc = could not find container \"22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea\": container with ID starting with 22d79a76c0440dc5323a35d7c79ca3a9e5734e972a869fa169a8ff146dc1b5ea not found: ID does not exist" Oct 30 22:24:29.883053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.883015 5036 scope.go:122] "RemoveContainer" containerID="5a9ddfea7e675b3029b4e1e89b26ea6e15139aad72bf0d556e3dd2a4a53e40e8" Oct 30 22:24:29.895336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.895314 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:24:29.900768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.900746 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-6c6bwhwpx"] Oct 30 22:24:29.911868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.911831 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:24:29.916495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:29.916471 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7f59t6d9g"] Oct 30 22:24:30.191767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:30.191730 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c066c74-fcd2-4796-93f5-20640daf4bdc" path="/var/lib/kubelet/pods/1c066c74-fcd2-4796-93f5-20640daf4bdc/volumes" Oct 30 22:24:30.192202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:30.192186 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234f8516-68cd-4216-8e00-fced9a946778" path="/var/lib/kubelet/pods/234f8516-68cd-4216-8e00-fced9a946778/volumes" Oct 30 22:24:32.323162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.323094 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:32.325484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.325278 5036 cpu_manager.go:414] "containerMap: removing container" podUID="482b83d4-e366-46cb-a75d-6b0522dc607f" containerName="revoke-keys" Oct 30 22:24:32.325672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.325654 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482b83d4-e366-46cb-a75d-6b0522dc607f" containerName="revoke-keys" Oct 30 22:24:32.325866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.325845 5036 cpu_manager.go:414] "containerMap: removing container" podUID="234f8516-68cd-4216-8e00-fced9a946778" containerName="payload-processing" Oct 30 22:24:32.326019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326005 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234f8516-68cd-4216-8e00-fced9a946778" containerName="payload-processing" Oct 30 22:24:32.326170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326156 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1c066c74-fcd2-4796-93f5-20640daf4bdc" containerName="payload-processing" Oct 30 22:24:32.326288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326256 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c066c74-fcd2-4796-93f5-20640daf4bdc" containerName="payload-processing" Oct 30 22:24:32.326389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326376 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd1990fc-e9c9-4ab2-b776-9a097950ce59" containerName="revoke-keys" Oct 30 22:24:32.326489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326477 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1990fc-e9c9-4ab2-b776-9a097950ce59" containerName="revoke-keys" Oct 30 22:24:32.326898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.326847 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="482b83d4-e366-46cb-a75d-6b0522dc607f" containerName="revoke-keys" Oct 30 22:24:32.327046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.327032 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="234f8516-68cd-4216-8e00-fced9a946778" containerName="payload-processing" Oct 30 22:24:32.327159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.327144 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c066c74-fcd2-4796-93f5-20640daf4bdc" containerName="payload-processing" Oct 30 22:24:32.327695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.327258 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1990fc-e9c9-4ab2-b776-9a097950ce59" containerName="revoke-keys" Oct 30 22:24:32.332782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.332759 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.352580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.352541 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:32.422089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.422053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2bj\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.422301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.422204 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.422359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.422297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.422398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.422385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.524315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.524249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2bj\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.524510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.524413 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.524510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.524465 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.524658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.524575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.525304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.525233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.525479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.525356 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.527555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.527530 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.538606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.538576 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2bj\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\") pod \"payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.677351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.677316 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:32.998247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.992335 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:33.005697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:32.999646 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.005697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.002548 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:33.115257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.114980 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfv65\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.116692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.115559 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.116692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.115745 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.116692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.115811 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217004 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217115 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfv65\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217860 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.217952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.217857 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.222108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.222082 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.230012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.229987 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfv65\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\") pod \"payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.327709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.327675 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:33.328240 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:33.328217 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70299f8a_2cb9_4ee6_831d_b1129f6c502a.slice/crio-2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee WatchSource:0}: Error finding container 2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee: Status 404 returned error can't find the container with id 2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee Oct 30 22:24:33.329175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.329154 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:33.640283 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:33.640225 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1058371c_d465_40dc_907e_654e5f601e85.slice/crio-cc8165b38cc24967ecbb8f1c9a1bbcc15ac632ba3d6a519cd62f03d98ff5e617 WatchSource:0}: Error finding container cc8165b38cc24967ecbb8f1c9a1bbcc15ac632ba3d6a519cd62f03d98ff5e617: Status 404 returned error can't find the container with id cc8165b38cc24967ecbb8f1c9a1bbcc15ac632ba3d6a519cd62f03d98ff5e617 Oct 30 22:24:33.640419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.640351 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:33.785809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.785768 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:33.796609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.796579 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:24:33.992034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.991941 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" event={"ID":"70299f8a-2cb9-4ee6-831d-b1129f6c502a","Type":"ContainerStarted","Data":"f33abd80c8ab7382b2f79c4c17f49bdda743fcf1438707bc09d3ec9e595d53c0"} Oct 30 22:24:33.992034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.992003 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" event={"ID":"70299f8a-2cb9-4ee6-831d-b1129f6c502a","Type":"ContainerStarted","Data":"2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee"} Oct 30 22:24:33.992034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.992032 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:33.992369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.992158 5036 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" secret="" err="secret \"payload-processing-external-model-e2e--2051f056-dockercfg-9hgtp\" not found" Oct 30 22:24:33.994096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.994065 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" event={"ID":"1058371c-d465-40dc-907e-654e5f601e85","Type":"ContainerStarted","Data":"9f59f86728e20d26fead6763082b3493c401be26190cbbf82a64f1ee53dfe89d"} Oct 30 22:24:33.994435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.994123 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" event={"ID":"1058371c-d465-40dc-907e-654e5f601e85","Type":"ContainerStarted","Data":"cc8165b38cc24967ecbb8f1c9a1bbcc15ac632ba3d6a519cd62f03d98ff5e617"} Oct 30 22:24:33.994435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:33.994153 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:34.025282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:34.025215 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" podStartSLOduration=2.025198162 podStartE2EDuration="2.025198162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:34.022081031 +0000 UTC m=+2697.972035977" watchObservedRunningTime="2026-10-30 22:24:34.025198162 +0000 UTC m=+2697.975153108" Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030331 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030403 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:34.53038313 +0000 UTC m=+2698.480338077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030420 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-e06d3afbed37: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030442 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030449 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a5f192/agc-cred-03ae22d0bf6c: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030471 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030524 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials podName:1058371c-d465-40dc-907e-654e5f601e85 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:34.530506867 +0000 UTC m=+2698.480461805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" (UID: "1058371c-d465-40dc-907e-654e5f601e85") : secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.030846 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.030813 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:34.530792524 +0000 UTC m=+2698.480747450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:34.365345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:34.362584 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" podStartSLOduration=2.3625610249999998 podStartE2EDuration="2.362561025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:34.062652069 +0000 UTC m=+2698.012607012" watchObservedRunningTime="2026-10-30 22:24:34.362561025 +0000 UTC m=+2698.312515973" Oct 30 22:24:34.365345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:34.363226 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:34.555244 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.555098 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556179 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:35.556130581 +0000 UTC m=+2699.506085512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556569 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a5f192/agc-cred-03ae22d0bf6c: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556585 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556630 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials podName:1058371c-d465-40dc-907e-654e5f601e85 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:35.556617526 +0000 UTC m=+2699.506572465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" (UID: "1058371c-d465-40dc-907e-654e5f601e85") : secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556683 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-e06d3afbed37: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556692 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:34.676142 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:34.556728 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:35.556716346 +0000 UTC m=+2699.506671276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:35.016970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:35.016327 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:35.479301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:35.476207 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:35.504284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:35.495298 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:24:35.619812 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619778 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-e06d3afbed37: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:35.619812 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619808 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:35.619812 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619809 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a5f192/agc-cred-03ae22d0bf6c: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:35.620154 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619828 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:35.620154 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619874 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:37.619852266 +0000 UTC m=+2701.569807205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:35.620154 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619892 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:35.620154 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619919 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials podName:1058371c-d465-40dc-907e-654e5f601e85 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:37.619898735 +0000 UTC m=+2701.569853665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" (UID: "1058371c-d465-40dc-907e-654e5f601e85") : secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:35.620154 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:35.619946 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:37.619935455 +0000 UTC m=+2701.569890378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:36.042285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:36.042228 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt"] Oct 30 22:24:36.048571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:36.048540 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c6fec3-d49qt"] Oct 30 22:24:36.192447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:36.192413 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4f9825-23ec-4c66-a47d-3f4b66e90c5a" path="/var/lib/kubelet/pods/5e4f9825-23ec-4c66-a47d-3f4b66e90c5a/volumes" Oct 30 22:24:37.216194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:37.216159 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:24:37.216817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:37.216787 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:24:37.701455 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701419 5036 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-plugins-e2e-extmod-a-b9d868: configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:37.701455 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701451 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b9d868/agc-cred-e06d3afbed37: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701471 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8: secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701504 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:41.701485555 +0000 UTC m=+2705.651440493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-b9d868" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701511 5036 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a5f192/agc-cred-03ae22d0bf6c: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701525 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials podName:70299f8a-2cb9-4ee6-831d-b1129f6c502a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:41.701516076 +0000 UTC m=+2705.651471030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a") : secret "agc-cred-e06d3afbed37" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701527 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc: secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:37.701745 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:37.701566 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials podName:1058371c-d465-40dc-907e-654e5f601e85 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:41.701553313 +0000 UTC m=+2705.651508244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" (UID: "1058371c-d465-40dc-907e-654e5f601e85") : secret "agc-cred-03ae22d0bf6c" not found Oct 30 22:24:38.797763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:38.797722 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" podUID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" containerName="payload-processing" containerID="cri-o://481940e0d1ca697e9fc0b7c161f176ff63f3fd8dcba5b7debd4bc078ed983a17" gracePeriod=25 Oct 30 22:24:39.142568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.142523 5036 generic.go:356] "Generic (PLEG): container finished" podID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" containerID="481940e0d1ca697e9fc0b7c161f176ff63f3fd8dcba5b7debd4bc078ed983a17" exitCode=0 Oct 30 22:24:39.142778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.142626 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" event={"ID":"1876bc34-eeac-4a72-9c72-b8ac23e89d82","Type":"ContainerDied","Data":"481940e0d1ca697e9fc0b7c161f176ff63f3fd8dcba5b7debd4bc078ed983a17"} Oct 30 22:24:39.184393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.184366 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:24:39.231713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.231680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\") pod \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " Oct 30 22:24:39.231897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.231805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\") pod \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " Oct 30 22:24:39.231972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.231951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\") pod \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " Oct 30 22:24:39.232138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.232123 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\") pod \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\" (UID: \"1876bc34-eeac-4a72-9c72-b8ac23e89d82\") " Oct 30 22:24:39.232217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.232144 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume" pod "1876bc34-eeac-4a72-9c72-b8ac23e89d82" (UID: "1876bc34-eeac-4a72-9c72-b8ac23e89d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:39.232217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.232155 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume" pod "1876bc34-eeac-4a72-9c72-b8ac23e89d82" (UID: "1876bc34-eeac-4a72-9c72-b8ac23e89d82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:39.233122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.233093 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.233252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.233124 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1876bc34-eeac-4a72-9c72-b8ac23e89d82-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.234670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.234640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials" pod "1876bc34-eeac-4a72-9c72-b8ac23e89d82" (UID: "1876bc34-eeac-4a72-9c72-b8ac23e89d82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:39.234779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.234716 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb" pod "1876bc34-eeac-4a72-9c72-b8ac23e89d82" (UID: "1876bc34-eeac-4a72-9c72-b8ac23e89d82"). InnerVolumeSpecName "kube-api-access-fr4cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:39.334925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.334890 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr4cb\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-kube-api-access-fr4cb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.334925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.334921 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1876bc34-eeac-4a72-9c72-b8ac23e89d82-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.404823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.404721 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" podUID="12985313-8ab2-42c6-be83-565725236808" containerName="payload-processing" containerID="cri-o://7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75" gracePeriod=25 Oct 30 22:24:39.701256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.701226 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:39.757932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.751116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\") pod \"12985313-8ab2-42c6-be83-565725236808\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " Oct 30 22:24:39.757932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.751300 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\") pod \"12985313-8ab2-42c6-be83-565725236808\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " Oct 30 22:24:39.757932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.751360 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\") pod \"12985313-8ab2-42c6-be83-565725236808\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " Oct 30 22:24:39.757932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.751419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\") pod \"12985313-8ab2-42c6-be83-565725236808\" (UID: \"12985313-8ab2-42c6-be83-565725236808\") " Oct 30 22:24:39.764194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.762499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume" pod "12985313-8ab2-42c6-be83-565725236808" (UID: "12985313-8ab2-42c6-be83-565725236808"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:39.765166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.765135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume" pod "12985313-8ab2-42c6-be83-565725236808" (UID: "12985313-8ab2-42c6-be83-565725236808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:39.786299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.782734 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials" pod "12985313-8ab2-42c6-be83-565725236808" (UID: "12985313-8ab2-42c6-be83-565725236808"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:39.818508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.818461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv" pod "12985313-8ab2-42c6-be83-565725236808" (UID: "12985313-8ab2-42c6-be83-565725236808"). InnerVolumeSpecName "kube-api-access-nwtfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:39.854618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.854581 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.854767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.854622 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtfv\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-kube-api-access-nwtfv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.854767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.854639 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12985313-8ab2-42c6-be83-565725236808-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.854767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:39.854655 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12985313-8ab2-42c6-be83-565725236808-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.016088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.016008 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" podUID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" containerName="payload-processing" containerID="cri-o://f33abd80c8ab7382b2f79c4c17f49bdda743fcf1438707bc09d3ec9e595d53c0" gracePeriod=25 Oct 30 22:24:40.017510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.017480 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" podUID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" containerName="payload-processing" containerID="cri-o://60acfa59a77fe24d6ac64efa3f368fb9c28f5cd1ee255008091ac9237420b43f" gracePeriod=25 Oct 30 22:24:40.156025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.155996 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" Oct 30 22:24:40.159561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.159517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k" event={"ID":"1876bc34-eeac-4a72-9c72-b8ac23e89d82","Type":"ContainerDied","Data":"3caf22c2ae911d206d9d1a51f30efeb68f972965407a5965f497257c60742f7f"} Oct 30 22:24:40.159793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.159597 5036 scope.go:122] "RemoveContainer" containerID="481940e0d1ca697e9fc0b7c161f176ff63f3fd8dcba5b7debd4bc078ed983a17" Oct 30 22:24:40.161621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.161578 5036 generic.go:356] "Generic (PLEG): container finished" podID="12985313-8ab2-42c6-be83-565725236808" containerID="7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75" exitCode=0 Oct 30 22:24:40.161729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.161689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" event={"ID":"12985313-8ab2-42c6-be83-565725236808","Type":"ContainerDied","Data":"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75"} Oct 30 22:24:40.161815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.161729 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" event={"ID":"12985313-8ab2-42c6-be83-565725236808","Type":"ContainerDied","Data":"2dafc19dc608c09b8293a34c90f0b749751c54426904a1c59ead04b41c9e6830"} Oct 30 22:24:40.161815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.161810 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv" Oct 30 22:24:40.168362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.167790 5036 generic.go:356] "Generic (PLEG): container finished" podID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" containerID="60acfa59a77fe24d6ac64efa3f368fb9c28f5cd1ee255008091ac9237420b43f" exitCode=0 Oct 30 22:24:40.168362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.167927 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" event={"ID":"014e830d-a7a0-4caa-862f-a76bec6f5bc3","Type":"ContainerDied","Data":"60acfa59a77fe24d6ac64efa3f368fb9c28f5cd1ee255008091ac9237420b43f"} Oct 30 22:24:40.175020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.174992 5036 generic.go:356] "Generic (PLEG): container finished" podID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" containerID="f33abd80c8ab7382b2f79c4c17f49bdda743fcf1438707bc09d3ec9e595d53c0" exitCode=0 Oct 30 22:24:40.175131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.175045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" event={"ID":"70299f8a-2cb9-4ee6-831d-b1129f6c502a","Type":"ContainerDied","Data":"f33abd80c8ab7382b2f79c4c17f49bdda743fcf1438707bc09d3ec9e595d53c0"} Oct 30 22:24:40.189547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.189522 5036 scope.go:122] "RemoveContainer" containerID="7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75" Oct 30 22:24:40.204200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.204179 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:24:40.205152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.205105 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-589d7765k"] Oct 30 22:24:40.218691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.218664 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:40.219747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.219718 5036 scope.go:122] "RemoveContainer" containerID="7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75" Oct 30 22:24:40.220134 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:40.220075 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75\": container with ID starting with 7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75 not found: ID does not exist" containerID="7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75" Oct 30 22:24:40.220134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.220102 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75"} err="failed to get container status \"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75\": rpc error: code = NotFound desc = could not find container \"7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75\": container with ID starting with 7384e6b41ae0fa8fd1bb06507d8759d3a596f84cf7eb6c39b0a94c64c5141b75 not found: ID does not exist" Oct 30 22:24:40.227717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.227697 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1a5bdf/payload-processing-external-model-e2e-extmod-a-1a5bdf-7df6smcmv"] Oct 30 22:24:40.354693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.354646 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:40.361907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.361881 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:40.463934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.463894 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\") pod \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " Oct 30 22:24:40.464164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464003 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\") pod \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " Oct 30 22:24:40.464164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464096 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\") pod \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " Oct 30 22:24:40.464351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464327 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\") pod \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " Oct 30 22:24:40.464426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464374 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\") pod \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " Oct 30 22:24:40.464426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\") pod \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\" (UID: \"70299f8a-2cb9-4ee6-831d-b1129f6c502a\") " Oct 30 22:24:40.464548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\") pod \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " Oct 30 22:24:40.464605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.464584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\") pod \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\" (UID: \"014e830d-a7a0-4caa-862f-a76bec6f5bc3\") " Oct 30 22:24:40.468779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.468731 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume" pod "014e830d-a7a0-4caa-862f-a76bec6f5bc3" (UID: "014e830d-a7a0-4caa-862f-a76bec6f5bc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.469172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.469145 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume" pod "70299f8a-2cb9-4ee6-831d-b1129f6c502a" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.477377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.470715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume" pod "70299f8a-2cb9-4ee6-831d-b1129f6c502a" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.477377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.470799 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume" pod "014e830d-a7a0-4caa-862f-a76bec6f5bc3" (UID: "014e830d-a7a0-4caa-862f-a76bec6f5bc3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.477377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.475096 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials" pod "014e830d-a7a0-4caa-862f-a76bec6f5bc3" (UID: "014e830d-a7a0-4caa-862f-a76bec6f5bc3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.478054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.478012 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g" pod "014e830d-a7a0-4caa-862f-a76bec6f5bc3" (UID: "014e830d-a7a0-4caa-862f-a76bec6f5bc3"). InnerVolumeSpecName "kube-api-access-8xj6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.479522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.479497 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj" pod "70299f8a-2cb9-4ee6-831d-b1129f6c502a" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a"). InnerVolumeSpecName "kube-api-access-6c2bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.481790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.481759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials" pod "70299f8a-2cb9-4ee6-831d-b1129f6c502a" (UID: "70299f8a-2cb9-4ee6-831d-b1129f6c502a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.495813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.495779 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" podUID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" containerName="payload-processing" containerID="cri-o://035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37" gracePeriod=25 Oct 30 22:24:40.570196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570160 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570194 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570209 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570223 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2bj\" (UniqueName: \"kubernetes.io/projected/70299f8a-2cb9-4ee6-831d-b1129f6c502a-kube-api-access-6c2bj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570238 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570250 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014e830d-a7a0-4caa-862f-a76bec6f5bc3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570282 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70299f8a-2cb9-4ee6-831d-b1129f6c502a-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.570447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.570297 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj6g\" (UniqueName: \"kubernetes.io/projected/014e830d-a7a0-4caa-862f-a76bec6f5bc3-kube-api-access-8xj6g\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.771360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.771334 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:24:40.874804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.874734 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\") pod \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " Oct 30 22:24:40.875193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.874944 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\") pod \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " Oct 30 22:24:40.875193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.874978 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\") pod \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " Oct 30 22:24:40.875193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.875072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume" pod "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" (UID: "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.875193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.875115 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\") pod \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\" (UID: \"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609\") " Oct 30 22:24:40.875474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.875431 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume" pod "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" (UID: "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.875830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.875814 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.875880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.875834 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.877491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.877469 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials" pod "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" (UID: "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.877605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.877522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv" pod "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" (UID: "8a6fa369-5cc9-409b-b6fb-eb3bf82ae609"). InnerVolumeSpecName "kube-api-access-x2gcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.977524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.977495 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gcv\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-kube-api-access-x2gcv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.977524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:40.977523 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.037705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.037660 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" podUID="1058371c-d465-40dc-907e-654e5f601e85" containerName="payload-processing" containerID="cri-o://9f59f86728e20d26fead6763082b3493c401be26190cbbf82a64f1ee53dfe89d" gracePeriod=25 Oct 30 22:24:41.189070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.189039 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" Oct 30 22:24:41.189224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.189062 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69" event={"ID":"014e830d-a7a0-4caa-862f-a76bec6f5bc3","Type":"ContainerDied","Data":"1a6a710e9802ff0c2457534e81c5fa2f5a98388a8e961780cdf86aab8c7cbcc7"} Oct 30 22:24:41.189224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.189136 5036 scope.go:122] "RemoveContainer" containerID="60acfa59a77fe24d6ac64efa3f368fb9c28f5cd1ee255008091ac9237420b43f" Oct 30 22:24:41.191038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.191013 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" containerID="035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37" exitCode=0 Oct 30 22:24:41.191390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.191135 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" Oct 30 22:24:41.191390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.191230 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" event={"ID":"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609","Type":"ContainerDied","Data":"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37"} Oct 30 22:24:41.191390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.191334 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j" event={"ID":"8a6fa369-5cc9-409b-b6fb-eb3bf82ae609","Type":"ContainerDied","Data":"f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513"} Oct 30 22:24:41.197304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.193960 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" Oct 30 22:24:41.197304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.194005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8" event={"ID":"70299f8a-2cb9-4ee6-831d-b1129f6c502a","Type":"ContainerDied","Data":"2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee"} Oct 30 22:24:41.198173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.197485 5036 generic.go:356] "Generic (PLEG): container finished" podID="1058371c-d465-40dc-907e-654e5f601e85" containerID="9f59f86728e20d26fead6763082b3493c401be26190cbbf82a64f1ee53dfe89d" exitCode=0 Oct 30 22:24:41.198173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.197548 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" event={"ID":"1058371c-d465-40dc-907e-654e5f601e85","Type":"ContainerDied","Data":"9f59f86728e20d26fead6763082b3493c401be26190cbbf82a64f1ee53dfe89d"} Oct 30 22:24:41.210782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.210759 5036 scope.go:122] "RemoveContainer" containerID="035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37" Oct 30 22:24:41.232429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.232406 5036 scope.go:122] "RemoveContainer" containerID="035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37" Oct 30 22:24:41.232987 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:41.232959 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37\": container with ID starting with 035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37 not found: ID does not exist" containerID="035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37" Oct 30 22:24:41.233101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.233001 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37"} err="failed to get container status \"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37\": rpc error: code = NotFound desc = could not find container \"035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37\": container with ID starting with 035fc3450271b049558ad17f392af458e2a6c23ebafa418589e94828ba43cd37 not found: ID does not exist" Oct 30 22:24:41.233101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.233028 5036 scope.go:122] "RemoveContainer" containerID="f33abd80c8ab7382b2f79c4c17f49bdda743fcf1438707bc09d3ec9e595d53c0" Oct 30 22:24:41.235511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.235480 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:41.246803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.246777 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-7877n2l69"] Oct 30 22:24:41.255621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.255601 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:24:41.261968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.261947 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-74f79vf8j"] Oct 30 22:24:41.268138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.268120 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:41.274159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.274138 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9d868/payload-processing-external-model-e2e-extmod-a-b9d868-649cwnsf8"] Oct 30 22:24:41.293820 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:41.293777 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6fa369_5cc9_409b_b6fb_eb3bf82ae609.slice/crio-f29c6758db857d1085dd5281412ca5d51f8a9d01e0f71d36d1942a7ad24a9513\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014e830d_a7a0_4caa_862f_a76bec6f5bc3.slice/crio-1a6a710e9802ff0c2457534e81c5fa2f5a98388a8e961780cdf86aab8c7cbcc7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70299f8a_2cb9_4ee6_831d_b1129f6c502a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014e830d_a7a0_4caa_862f_a76bec6f5bc3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70299f8a_2cb9_4ee6_831d_b1129f6c502a.slice/crio-2d122231994ccf1917c723f48a1a5c5441e0aae4ec03c268398afe401ef140ee\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6fa369_5cc9_409b_b6fb_eb3bf82ae609.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:24:41.394179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.394154 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:41.484240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\") pod \"1058371c-d465-40dc-907e-654e5f601e85\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " Oct 30 22:24:41.484240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\") pod \"1058371c-d465-40dc-907e-654e5f601e85\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " Oct 30 22:24:41.484453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\") pod \"1058371c-d465-40dc-907e-654e5f601e85\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " Oct 30 22:24:41.484453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\") pod \"1058371c-d465-40dc-907e-654e5f601e85\" (UID: \"1058371c-d465-40dc-907e-654e5f601e85\") " Oct 30 22:24:41.484671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484646 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume" pod "1058371c-d465-40dc-907e-654e5f601e85" (UID: "1058371c-d465-40dc-907e-654e5f601e85"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:41.484711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.484680 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume" pod "1058371c-d465-40dc-907e-654e5f601e85" (UID: "1058371c-d465-40dc-907e-654e5f601e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:41.485231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.485210 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.485347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.485238 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1058371c-d465-40dc-907e-654e5f601e85-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.487094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.487067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials" pod "1058371c-d465-40dc-907e-654e5f601e85" (UID: "1058371c-d465-40dc-907e-654e5f601e85"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:41.487905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.487637 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65" pod "1058371c-d465-40dc-907e-654e5f601e85" (UID: "1058371c-d465-40dc-907e-654e5f601e85"). InnerVolumeSpecName "kube-api-access-gfv65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:41.587038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.587004 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfv65\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-kube-api-access-gfv65\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.587038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:41.587031 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1058371c-d465-40dc-907e-654e5f601e85-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:42.190170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.190131 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" path="/var/lib/kubelet/pods/014e830d-a7a0-4caa-862f-a76bec6f5bc3/volumes" Oct 30 22:24:42.190629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.190615 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12985313-8ab2-42c6-be83-565725236808" path="/var/lib/kubelet/pods/12985313-8ab2-42c6-be83-565725236808/volumes" Oct 30 22:24:42.191000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.190987 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" path="/var/lib/kubelet/pods/1876bc34-eeac-4a72-9c72-b8ac23e89d82/volumes" Oct 30 22:24:42.191403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.191389 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" path="/var/lib/kubelet/pods/70299f8a-2cb9-4ee6-831d-b1129f6c502a/volumes" Oct 30 22:24:42.192061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.192046 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" path="/var/lib/kubelet/pods/8a6fa369-5cc9-409b-b6fb-eb3bf82ae609/volumes" Oct 30 22:24:42.209477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.209456 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" Oct 30 22:24:42.209598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.209479 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc" event={"ID":"1058371c-d465-40dc-907e-654e5f601e85","Type":"ContainerDied","Data":"cc8165b38cc24967ecbb8f1c9a1bbcc15ac632ba3d6a519cd62f03d98ff5e617"} Oct 30 22:24:42.209598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.209535 5036 scope.go:122] "RemoveContainer" containerID="9f59f86728e20d26fead6763082b3493c401be26190cbbf82a64f1ee53dfe89d" Oct 30 22:24:42.230579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.230549 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:42.235341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.235316 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5f192/payload-processing-external-model-e2e-extmod-b-a5f192-5876852kc"] Oct 30 22:24:42.625858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.625775 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:24:42.626570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626553 5036 cpu_manager.go:414] "containerMap: removing container" podUID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" containerName="payload-processing" Oct 30 22:24:42.626570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626570 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626583 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626588 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626614 5036 cpu_manager.go:414] "containerMap: removing container" podUID="12985313-8ab2-42c6-be83-565725236808" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626619 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12985313-8ab2-42c6-be83-565725236808" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626647 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1058371c-d465-40dc-907e-654e5f601e85" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626652 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1058371c-d465-40dc-907e-654e5f601e85" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626663 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626667 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626679 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" containerName="payload-processing" Oct 30 22:24:42.626728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626685 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626789 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70299f8a-2cb9-4ee6-831d-b1129f6c502a" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626807 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="014e830d-a7a0-4caa-862f-a76bec6f5bc3" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626823 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6fa369-5cc9-409b-b6fb-eb3bf82ae609" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626837 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="12985313-8ab2-42c6-be83-565725236808" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626852 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1058371c-d465-40dc-907e-654e5f601e85" containerName="payload-processing" Oct 30 22:24:42.627239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.626862 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1876bc34-eeac-4a72-9c72-b8ac23e89d82" containerName="payload-processing" Oct 30 22:24:42.633710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.633687 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.640598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.637984 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-120cb1-openshift-default-dockercfg-9p94d\"" Oct 30 22:24:42.642338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.642310 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:24:42.729346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729312 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729532 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729454 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729533 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729720 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729749 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.729947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.729913 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2478z\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.831843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.831782 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.831867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.831904 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.831932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.831960 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2478z\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.832133 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.832253 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.832349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.832404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.832905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.832876 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.842306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.833493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.842306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.834748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.842306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.835597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.842306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.835875 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.871789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.869990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.897747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.890579 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.897747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.891142 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.897747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.895793 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2478z\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:42.973293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:42.972231 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:43.443882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.443853 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:24:43.444884 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:43.444857 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ecec29_954d_4306_98fe_62d3e2085f81.slice/crio-2ae2775dfee94a5d5f04092be475f3fb777b1bffd985ee3d1fe9c5d786e6f8e0 WatchSource:0}: Error finding container 2ae2775dfee94a5d5f04092be475f3fb777b1bffd985ee3d1fe9c5d786e6f8e0: Status 404 returned error can't find the container with id 2ae2775dfee94a5d5f04092be475f3fb777b1bffd985ee3d1fe9c5d786e6f8e0 Oct 30 22:24:43.448369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.448326 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:43.448497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.448420 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:43.448497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.448469 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:43.665192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.665154 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:24:43.731649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.731571 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:24:43.731947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.731876 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.740952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.740920 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-df200a-openshift-default-dockercfg-rrfqx\"" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862312 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862382 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862434 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862468 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpfz\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862606 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862724 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:43.933626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.862801 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.966338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990439 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990481 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990760 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990819 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.990919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpfz\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.991091 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.991632 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.991899 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.992130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.995466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.009607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:43.996851 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.103667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.103630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpfz\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.114118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.114046 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.114320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.114212 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.125853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.124956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.191035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.191004 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1058371c-d465-40dc-907e-654e5f601e85" path="/var/lib/kubelet/pods/1058371c-d465-40dc-907e-654e5f601e85/volumes" Oct 30 22:24:44.250312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.250080 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerStarted","Data":"977b69c03187d6938e9460990fc7d13b2bbdefa1d64d63622c7882324b8cc88b"} Oct 30 22:24:44.250312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.250154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerStarted","Data":"2ae2775dfee94a5d5f04092be475f3fb777b1bffd985ee3d1fe9c5d786e6f8e0"} Oct 30 22:24:44.280728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.280626 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podStartSLOduration=2.2806106919999998 podStartE2EDuration="2.280610692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:44.278939435 +0000 UTC m=+2708.228894381" watchObservedRunningTime="2026-10-30 22:24:44.280610692 +0000 UTC m=+2708.230565635" Oct 30 22:24:44.389618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.389586 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:44.780832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.780802 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:24:44.781338 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:44.781310 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a83fd3d_d397_4f39_b2cc_8ae7feec6ade.slice/crio-6211f3512ccacda2a4805f7204e441efa77afaa4d38f0c8b2992ed104f214ee9 WatchSource:0}: Error finding container 6211f3512ccacda2a4805f7204e441efa77afaa4d38f0c8b2992ed104f214ee9: Status 404 returned error can't find the container with id 6211f3512ccacda2a4805f7204e441efa77afaa4d38f0c8b2992ed104f214ee9 Oct 30 22:24:44.784523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.784490 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:44.784599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.784557 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:44.784599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.784586 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:44.973027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.972984 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:44.980015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:44.979991 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:45.263434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.263399 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerStarted","Data":"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354"} Oct 30 22:24:45.263639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.263482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerStarted","Data":"6211f3512ccacda2a4805f7204e441efa77afaa4d38f0c8b2992ed104f214ee9"} Oct 30 22:24:45.263799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.263777 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:45.264810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.264786 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:24:45.289433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.289362 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podStartSLOduration=2.2893430759999998 podStartE2EDuration="2.289343076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:45.287999411 +0000 UTC m=+2709.237954360" watchObservedRunningTime="2026-10-30 22:24:45.289343076 +0000 UTC m=+2709.239298042" Oct 30 22:24:45.389877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.389842 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:45.396335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:45.396308 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:46.272806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:46.272770 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:46.273607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:46.273589 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:24:48.008422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.008390 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:24:48.013784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.013763 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.018529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.018507 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-120cb1\"" Oct 30 22:24:48.030128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.030104 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:24:48.036375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.035830 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-120cb1-dockercfg-ck69x\"" Oct 30 22:24:48.036938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.036911 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:24:48.059322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.047475 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.076278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.076231 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:24:48.112627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.104020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.112627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.104257 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8d2\" (UniqueName: \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.456258 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:24:48.455236 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:24:48.478100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.458779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.478100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.458885 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8d2\" (UniqueName: \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.478100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.458916 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.478100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.458983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9gx\" (UniqueName: \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.478100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.466547 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.507309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.506563 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8d2\" (UniqueName: \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\") pod \"payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.576898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.576859 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.577084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.576916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9gx\" (UniqueName: \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.577805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.577784 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.589222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.589199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9gx\" (UniqueName: \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\") pod \"payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.639604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.639556 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:48.675852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.675822 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:48.930607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.930572 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:24:48.938209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.938181 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:48.952986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.952961 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-df200a-dockercfg-64bxh\"" Oct 30 22:24:48.953312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.953246 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-df200a\"" Oct 30 22:24:48.955133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.955022 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:24:48.966599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.966570 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:24:48.985896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.985867 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:24:48.987092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.987053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:48.987453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.987429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdg6h\" (UniqueName: \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:48.995741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.995716 5036 scope.go:122] "RemoveContainer" containerID="ac4d6ed6dab5130eed846fb89f86eee1da4c93a7ebd131eeefecdfdcbe4569bf" Oct 30 22:24:48.996326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:48.996307 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.005432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.005143 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:24:49.012114 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:49.012083 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70097ceb_7814_42d1_a94a_d269decc53ac.slice/crio-c1bf1ea34764eac508d0fb4c0f127e0c120b21976ca614ddd7f1da4658e36c9e WatchSource:0}: Error finding container c1bf1ea34764eac508d0fb4c0f127e0c120b21976ca614ddd7f1da4658e36c9e: Status 404 returned error can't find the container with id c1bf1ea34764eac508d0fb4c0f127e0c120b21976ca614ddd7f1da4658e36c9e Oct 30 22:24:49.013898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.013067 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:24:49.029583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.029493 5036 scope.go:122] "RemoveContainer" containerID="3ca7c8d4c9bbfacf0122c1db6e858fb4d32bfb42e4365638f086060373764aff" Oct 30 22:24:49.108311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.089559 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdg6h\" (UniqueName: \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:49.108311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.089648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lpg\" (UniqueName: \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.108311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.089685 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.108311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.089762 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:49.108311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.090702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:49.335304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.334807 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lpg\" (UniqueName: \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.335304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.334856 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.343544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.336123 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.392543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.385183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdg6h\" (UniqueName: \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\") pod \"payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:49.407020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.406975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lpg\" (UniqueName: \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\") pod \"payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.502399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.502369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" event={"ID":"70097ceb-7814-42d1-a94a-d269decc53ac","Type":"ContainerStarted","Data":"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe"} Oct 30 22:24:49.502489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.502434 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" event={"ID":"70097ceb-7814-42d1-a94a-d269decc53ac","Type":"ContainerStarted","Data":"c1bf1ea34764eac508d0fb4c0f127e0c120b21976ca614ddd7f1da4658e36c9e"} Oct 30 22:24:49.504529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.504503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" event={"ID":"5454cfbd-cd38-401e-a469-7655539d3e53","Type":"ContainerStarted","Data":"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190"} Oct 30 22:24:49.504651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.504554 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" event={"ID":"5454cfbd-cd38-401e-a469-7655539d3e53","Type":"ContainerStarted","Data":"a28f2cb788bd5323029df1681f6eb2ebe45c5aaa91e3660e468c65454499087d"} Oct 30 22:24:49.504651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.504612 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:24:49.528989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.528945 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" podStartSLOduration=2.528930289 podStartE2EDuration="2.528930289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:49.52405059 +0000 UTC m=+2713.474005547" watchObservedRunningTime="2026-10-30 22:24:49.528930289 +0000 UTC m=+2713.478885245" Oct 30 22:24:49.561066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.561037 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:49.624733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.624506 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:49.706661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.706625 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd"] Oct 30 22:24:49.716608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.716581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.722329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.721442 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:49.722329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.722100 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:24:49.722329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.722169 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:49.722329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.722247 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:49.751484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.751318 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd"] Oct 30 22:24:49.848371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.848337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bdx\" (UniqueName: \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.848650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.848629 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.848886 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.848871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.973288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.970633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.973288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.970935 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bdx\" (UniqueName: \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.973288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.970988 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.988941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.988911 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.992395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.992343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bdx\" (UniqueName: \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:49.993251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:49.993209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8be69cb-nhwtd\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:50.032256 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:50.032220 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe41384f_2bc1_4094_a410_6aa343541589.slice/crio-6f171725a7361c6b8460ad39d1d3e681b260f02446e7714f5e1d2721581ab64c WatchSource:0}: Error finding container 6f171725a7361c6b8460ad39d1d3e681b260f02446e7714f5e1d2721581ab64c: Status 404 returned error can't find the container with id 6f171725a7361c6b8460ad39d1d3e681b260f02446e7714f5e1d2721581ab64c Oct 30 22:24:50.032754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.032471 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:24:50.074446 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:50.074410 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d657993_9a1e_4f43_b331_7d758a6245ab.slice/crio-e6dae9d49e944a933019c791651b14568598f9a52a23ec45105f0efa17676276 WatchSource:0}: Error finding container e6dae9d49e944a933019c791651b14568598f9a52a23ec45105f0efa17676276: Status 404 returned error can't find the container with id e6dae9d49e944a933019c791651b14568598f9a52a23ec45105f0efa17676276 Oct 30 22:24:50.074582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.074549 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:24:50.099328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.099299 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:50.114319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.114077 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8"] Oct 30 22:24:50.121665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.121643 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.131961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.131934 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8"] Oct 30 22:24:50.281074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.280855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.281074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.280965 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hg9r\" (UniqueName: \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.281278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.281094 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.382882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.382846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.383075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.382952 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hg9r\" (UniqueName: \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.383075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.382996 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.383686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.383664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.385506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.385484 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.394835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.394812 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hg9r\" (UniqueName: \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\") pod \"maas-api-revoke-sub-0e813706-p9mm8\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.437878 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:50.437847 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedf517e_42fd_4ef3_b03d_9a097f301ed5.slice/crio-35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81 WatchSource:0}: Error finding container 35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81: Status 404 returned error can't find the container with id 35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81 Oct 30 22:24:50.439292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.439249 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd"] Oct 30 22:24:50.486070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.486042 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:50.518230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.518160 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" event={"ID":"bedf517e-42fd-4ef3-b03d-9a097f301ed5","Type":"ContainerStarted","Data":"35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81"} Oct 30 22:24:50.521893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.521159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" event={"ID":"be41384f-2bc1-4094-a410-6aa343541589","Type":"ContainerStarted","Data":"ce3ddc7b4f439cf62a1275cec8468221e5f505a743802ec243b8758d9f043066"} Oct 30 22:24:50.521893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.521219 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" event={"ID":"be41384f-2bc1-4094-a410-6aa343541589","Type":"ContainerStarted","Data":"6f171725a7361c6b8460ad39d1d3e681b260f02446e7714f5e1d2721581ab64c"} Oct 30 22:24:50.521893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.521485 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:24:50.524973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.524950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" event={"ID":"3d657993-9a1e-4f43-b331-7d758a6245ab","Type":"ContainerStarted","Data":"2cf668901ce1be06836cd1fcdc64b620a5b429c8de633a9243b118a7accc778b"} Oct 30 22:24:50.525643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.525360 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:24:50.525643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.525395 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" event={"ID":"3d657993-9a1e-4f43-b331-7d758a6245ab","Type":"ContainerStarted","Data":"e6dae9d49e944a933019c791651b14568598f9a52a23ec45105f0efa17676276"} Oct 30 22:24:50.525643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.525424 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:24:50.542723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.542618 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" podStartSLOduration=2.5425997689999997 podStartE2EDuration="2.542599769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:50.539210121 +0000 UTC m=+2714.489165068" watchObservedRunningTime="2026-10-30 22:24:50.542599769 +0000 UTC m=+2714.492554716" Oct 30 22:24:50.567926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.567875 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" podStartSLOduration=2.5678606889999998 podStartE2EDuration="2.567860689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:50.561700698 +0000 UTC m=+2714.511655646" watchObservedRunningTime="2026-10-30 22:24:50.567860689 +0000 UTC m=+2714.517815651" Oct 30 22:24:50.580255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.580208 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" podStartSLOduration=2.58019459 podStartE2EDuration="2.58019459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:50.57841473 +0000 UTC m=+2714.528369676" watchObservedRunningTime="2026-10-30 22:24:50.58019459 +0000 UTC m=+2714.530149537" Oct 30 22:24:50.870132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:50.870101 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8"] Oct 30 22:24:50.914439 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:24:50.914408 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda44fe22_0efb_49d2_9309_6ef0577f03c1.slice/crio-a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256 WatchSource:0}: Error finding container a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256: Status 404 returned error can't find the container with id a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256 Oct 30 22:24:51.538139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:51.538029 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" event={"ID":"da44fe22-0efb-49d2-9309-6ef0577f03c1","Type":"ContainerStarted","Data":"4b663830632ebfb00b15b08437b0605ccec0e131709da4217b4bbb13a52c878a"} Oct 30 22:24:51.538139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:51.538109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" event={"ID":"da44fe22-0efb-49d2-9309-6ef0577f03c1","Type":"ContainerStarted","Data":"a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256"} Oct 30 22:24:51.540023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:51.539991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" event={"ID":"bedf517e-42fd-4ef3-b03d-9a097f301ed5","Type":"ContainerStarted","Data":"b3a98d6a65766ef2dd0ba47b61ae2d945044f8c7bf66470b9e357a7a979b798e"} Oct 30 22:24:51.565021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:51.564964 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" podStartSLOduration=1.564945504 podStartE2EDuration="1.564945504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:51.559101029 +0000 UTC m=+2715.509055974" watchObservedRunningTime="2026-10-30 22:24:51.564945504 +0000 UTC m=+2715.514900451" Oct 30 22:24:51.589678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:51.589628 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" podStartSLOduration=2.589615587 podStartE2EDuration="2.589615587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:51.583572595 +0000 UTC m=+2715.533527541" watchObservedRunningTime="2026-10-30 22:24:51.589615587 +0000 UTC m=+2715.539570533" Oct 30 22:24:52.635567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:52.635530 5036 generic.go:356] "Generic (PLEG): container finished" podID="da44fe22-0efb-49d2-9309-6ef0577f03c1" containerID="4b663830632ebfb00b15b08437b0605ccec0e131709da4217b4bbb13a52c878a" exitCode=0 Oct 30 22:24:52.636321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:52.636292 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" event={"ID":"da44fe22-0efb-49d2-9309-6ef0577f03c1","Type":"ContainerDied","Data":"4b663830632ebfb00b15b08437b0605ccec0e131709da4217b4bbb13a52c878a"} Oct 30 22:24:52.654768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:52.654730 5036 generic.go:356] "Generic (PLEG): container finished" podID="bedf517e-42fd-4ef3-b03d-9a097f301ed5" containerID="b3a98d6a65766ef2dd0ba47b61ae2d945044f8c7bf66470b9e357a7a979b798e" exitCode=0 Oct 30 22:24:52.655232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:52.655142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" event={"ID":"bedf517e-42fd-4ef3-b03d-9a097f301ed5","Type":"ContainerDied","Data":"b3a98d6a65766ef2dd0ba47b61ae2d945044f8c7bf66470b9e357a7a979b798e"} Oct 30 22:24:53.884168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.884137 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:53.967573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.961018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\") pod \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " Oct 30 22:24:53.967573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.961143 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\") pod \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " Oct 30 22:24:53.967573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.961174 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\" (UniqueName: \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\") pod \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\" (UID: \"bedf517e-42fd-4ef3-b03d-9a097f301ed5\") " Oct 30 22:24:53.967573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.961798 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca" pod "bedf517e-42fd-4ef3-b03d-9a097f301ed5" (UID: "bedf517e-42fd-4ef3-b03d-9a097f301ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:53.967573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.962189 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:53.981044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.981006 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca" pod "bedf517e-42fd-4ef3-b03d-9a097f301ed5" (UID: "bedf517e-42fd-4ef3-b03d-9a097f301ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:53.988527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:53.988488 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx" pod "bedf517e-42fd-4ef3-b03d-9a097f301ed5" (UID: "bedf517e-42fd-4ef3-b03d-9a097f301ed5"). InnerVolumeSpecName "kube-api-access-g5bdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:54.003205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.003178 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:54.063650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.063618 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedf517e-42fd-4ef3-b03d-9a097f301ed5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:54.063650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.063651 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bdx\" (UniqueName: \"kubernetes.io/projected/bedf517e-42fd-4ef3-b03d-9a097f301ed5-kube-api-access-g5bdx\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:54.164253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.164213 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\" (UniqueName: \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\") pod \"da44fe22-0efb-49d2-9309-6ef0577f03c1\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " Oct 30 22:24:54.164523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.164503 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\") pod \"da44fe22-0efb-49d2-9309-6ef0577f03c1\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " Oct 30 22:24:54.164769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.164742 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\") pod \"da44fe22-0efb-49d2-9309-6ef0577f03c1\" (UID: \"da44fe22-0efb-49d2-9309-6ef0577f03c1\") " Oct 30 22:24:54.164883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.164864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca" pod "da44fe22-0efb-49d2-9309-6ef0577f03c1" (UID: "da44fe22-0efb-49d2-9309-6ef0577f03c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:54.165704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.165677 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:54.166998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.166974 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r" pod "da44fe22-0efb-49d2-9309-6ef0577f03c1" (UID: "da44fe22-0efb-49d2-9309-6ef0577f03c1"). InnerVolumeSpecName "kube-api-access-5hg9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:54.167476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.167443 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca" pod "da44fe22-0efb-49d2-9309-6ef0577f03c1" (UID: "da44fe22-0efb-49d2-9309-6ef0577f03c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:54.268068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.267970 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da44fe22-0efb-49d2-9309-6ef0577f03c1-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:54.268068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.268013 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hg9r\" (UniqueName: \"kubernetes.io/projected/da44fe22-0efb-49d2-9309-6ef0577f03c1-kube-api-access-5hg9r\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:54.675927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.675899 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" Oct 30 22:24:54.675927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.675904 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8" event={"ID":"da44fe22-0efb-49d2-9309-6ef0577f03c1","Type":"ContainerDied","Data":"a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256"} Oct 30 22:24:54.676234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.675993 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61f3aebad32cec0256f8f4bc56aec3cdf74ce6910bd9c79cbdda938434fc256" Oct 30 22:24:54.677669 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.677651 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" Oct 30 22:24:54.677765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.677682 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd" event={"ID":"bedf517e-42fd-4ef3-b03d-9a097f301ed5","Type":"ContainerDied","Data":"35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81"} Oct 30 22:24:54.677765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:24:54.677731 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35852cb44f61b631fbea242958ae8f607f5bd443052a44925686514905023f81" Oct 30 22:25:00.528368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.528335 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.979541 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555"] Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.980852 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bedf517e-42fd-4ef3-b03d-9a097f301ed5" containerName="revoke-keys" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.980874 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedf517e-42fd-4ef3-b03d-9a097f301ed5" containerName="revoke-keys" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.980906 5036 cpu_manager.go:414] "containerMap: removing container" podUID="da44fe22-0efb-49d2-9309-6ef0577f03c1" containerName="revoke-keys" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.980914 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da44fe22-0efb-49d2-9309-6ef0577f03c1" containerName="revoke-keys" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.981098 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedf517e-42fd-4ef3-b03d-9a097f301ed5" containerName="revoke-keys" Oct 30 22:25:00.982012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:00.981112 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="da44fe22-0efb-49d2-9309-6ef0577f03c1" containerName="revoke-keys" Oct 30 22:25:01.019918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.015833 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555"] Oct 30 22:25:01.019918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.015956 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.102288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.102231 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.102471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.102449 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.102536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.102525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.102587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.102576 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6nz\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.224808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.224770 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.229294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.225287 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.229606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.229583 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.229781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.229766 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6nz\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.258344 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.228401 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-67105abf9904: secret "agc-cred-67105abf9904" not found Oct 30 22:25:01.258344 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.256325 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555: secret "agc-cred-67105abf9904" not found Oct 30 22:25:01.258344 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.256427 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials podName:ef41144b-b2aa-46f5-99f3-e0786fffe3a1 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:01.756400917 +0000 UTC m=+2725.706355843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials") pod "payload-processing-external-model-78bf7bfc87-hs555" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1") : secret "agc-cred-67105abf9904" not found Oct 30 22:25:01.258344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.232914 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.258344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.229333 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.265017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.264981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6nz\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.392079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.392038 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555"] Oct 30 22:25:01.392693 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.392665 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" podUID="ef41144b-b2aa-46f5-99f3-e0786fffe3a1" Oct 30 22:25:01.397747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.397716 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:25:01.544718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.544626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:25:01.545173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.544720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:25:01.545173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.544907 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:25:01.889371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:01.882477 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:01.889371 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.883059 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-67105abf9904: secret "agc-cred-67105abf9904" not found Oct 30 22:25:01.889371 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.883079 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555: secret "agc-cred-67105abf9904" not found Oct 30 22:25:01.889371 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:01.883141 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials podName:ef41144b-b2aa-46f5-99f3-e0786fffe3a1 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:02.883122149 +0000 UTC m=+2726.833077080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials") pod "payload-processing-external-model-78bf7bfc87-hs555" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1") : secret "agc-cred-67105abf9904" not found Oct 30 22:25:02.091294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.089957 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:02.130326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.128960 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.202526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\") pod \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.202767 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\") pod \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.202994 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\") pod \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.203433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume" pod "ef41144b-b2aa-46f5-99f3-e0786fffe3a1" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.203764 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume" pod "ef41144b-b2aa-46f5-99f3-e0786fffe3a1" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.203944 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:02.209334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.203958 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:02.210619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.210590 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz" pod "ef41144b-b2aa-46f5-99f3-e0786fffe3a1" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1"). InnerVolumeSpecName "kube-api-access-td6nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:02.305486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.305449 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6nz\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-kube-api-access-td6nz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:02.916065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:02.916030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials\") pod \"payload-processing-external-model-78bf7bfc87-hs555\" (UID: \"ef41144b-b2aa-46f5-99f3-e0786fffe3a1\") " pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:02.916544 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:02.916223 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-67105abf9904: secret "agc-cred-67105abf9904" not found Oct 30 22:25:02.916544 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:02.916245 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555: secret "agc-cred-67105abf9904" not found Oct 30 22:25:02.916544 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:02.916332 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials podName:ef41144b-b2aa-46f5-99f3-e0786fffe3a1 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:04.916315421 +0000 UTC m=+2728.866270358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials") pod "payload-processing-external-model-78bf7bfc87-hs555" (UID: "ef41144b-b2aa-46f5-99f3-e0786fffe3a1") : secret "agc-cred-67105abf9904" not found Oct 30 22:25:03.098310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.098281 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555" Oct 30 22:25:03.148713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.148635 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555"] Oct 30 22:25:03.159938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.159905 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf7bfc87-hs555"] Oct 30 22:25:03.232715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.230623 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef41144b-b2aa-46f5-99f3-e0786fffe3a1-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:03.247065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.247031 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:25:03.254206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.254185 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.264403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.264368 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28d39b-openshift-default-dockercfg-qsmx6\"" Oct 30 22:25:03.270728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.270699 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:25:03.332711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.332218 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.332711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.332411 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.332711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.332667 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.332958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.332805 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xf4\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.332958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.332863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.333097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.333032 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.333393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.333370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.333481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.333466 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.333523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.333512 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.435514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.435481 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.435680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.435536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xf4\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.435680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.435650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.435826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.435731 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.435950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.435933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436010 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436054 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436123 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436431 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.436608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.436582 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.439354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.439001 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.439354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.439218 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.461032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.460985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xf4\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.463356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.463331 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:03.575068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:03.574987 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:04.198443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.198356 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef41144b-b2aa-46f5-99f3-e0786fffe3a1" path="/var/lib/kubelet/pods/ef41144b-b2aa-46f5-99f3-e0786fffe3a1/volumes" Oct 30 22:25:04.284717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.284688 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:25:04.286971 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:25:04.286938 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98fec20_adfa_4090_a5ed_39d187302155.slice/crio-5c83203cede0f6f6aa4db7051bde5a87ef5615ab2daa4c1d36c2bbbefe507b40 WatchSource:0}: Error finding container 5c83203cede0f6f6aa4db7051bde5a87ef5615ab2daa4c1d36c2bbbefe507b40: Status 404 returned error can't find the container with id 5c83203cede0f6f6aa4db7051bde5a87ef5615ab2daa4c1d36c2bbbefe507b40 Oct 30 22:25:04.290805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.290761 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:04.290896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.290856 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:04.290936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.290907 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:04.367044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.367016 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc"] Oct 30 22:25:04.380038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.380015 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.384495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.384341 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-01dc4dac3e0d\"" Oct 30 22:25:04.384495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.384470 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-njd4r\"" Oct 30 22:25:04.385436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.385413 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc"] Oct 30 22:25:04.485876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.485785 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc"] Oct 30 22:25:04.490318 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.486403 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fdql5 plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" podUID="739f1389-f662-4586-ae5a-68aa31961d39" Oct 30 22:25:04.490318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.488530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.490318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.488584 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.490318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.488726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.490318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.488874 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.590438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.590398 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.590608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.590551 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.590714 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.590693 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-01dc4dac3e0d: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:04.590753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.590697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.590797 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.590716 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:04.590797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.590775 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.590862 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.590836 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:05.090817083 +0000 UTC m=+2729.040772006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:04.590912 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.590865 5036 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:04.590949 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.590929 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:05.090910753 +0000 UTC m=+2729.040865699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:04.591296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:04.591260 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:04.595275 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.595245 5036 projected.go:196] Error preparing data for projected volume kube-api-access-fdql5 for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:04.595355 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:04.595314 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5 podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:05.095298667 +0000 UTC m=+2729.045253593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fdql5" (UniqueName: "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:05.121612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.121576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:05.121839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.121699 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:05.121839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.121834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:05.121994 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.121943 5036 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:05.121994 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.121991 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:06.12197839 +0000 UTC m=+2730.071933313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:05.122122 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.121992 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-01dc4dac3e0d: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:05.122122 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.122012 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:05.122122 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.122065 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:06.122047647 +0000 UTC m=+2730.072002586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:05.123138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.123113 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerStarted","Data":"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95"} Oct 30 22:25:05.123317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.123164 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerStarted","Data":"5c83203cede0f6f6aa4db7051bde5a87ef5615ab2daa4c1d36c2bbbefe507b40"} Oct 30 22:25:05.123317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.123224 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:05.124870 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.124839 5036 projected.go:196] Error preparing data for projected volume kube-api-access-fdql5 for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:05.124988 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:05.124906 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5 podName:739f1389-f662-4586-ae5a-68aa31961d39 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:06.124887706 +0000 UTC m=+2730.074842644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdql5" (UniqueName: "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5") pod "payload-processing-external-model-768f4bb6dd-hb8sc" (UID: "739f1389-f662-4586-ae5a-68aa31961d39") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:05.133543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.133519 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:05.148717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.148657 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podStartSLOduration=2.148641304 podStartE2EDuration="2.148641304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:05.145044534 +0000 UTC m=+2729.094999480" watchObservedRunningTime="2026-10-30 22:25:05.148641304 +0000 UTC m=+2729.098596251" Oct 30 22:25:05.222649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.222617 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\") pod \"739f1389-f662-4586-ae5a-68aa31961d39\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " Oct 30 22:25:05.223230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.223192 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume" pod "739f1389-f662-4586-ae5a-68aa31961d39" (UID: "739f1389-f662-4586-ae5a-68aa31961d39"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:05.223811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.223783 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:05.582435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.581475 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:05.708289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:05.686629 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:06.368303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.368152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.368814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.368446 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.368814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.368700 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.456544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.448308 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" podUID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" containerName="payload-processing" containerID="cri-o://83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356" gracePeriod=25 Oct 30 22:25:06.526299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.523735 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.632022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.595630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.632022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.597082 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"payload-processing-external-model-768f4bb6dd-hb8sc\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.658417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.653380 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:06.658417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.653874 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:25:06.786089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.786059 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") pod \"739f1389-f662-4586-ae5a-68aa31961d39\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " Oct 30 22:25:06.786642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.786619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") pod \"739f1389-f662-4586-ae5a-68aa31961d39\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " Oct 30 22:25:06.786889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.786850 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") pod \"739f1389-f662-4586-ae5a-68aa31961d39\" (UID: \"739f1389-f662-4586-ae5a-68aa31961d39\") " Oct 30 22:25:06.787192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.787164 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume" pod "739f1389-f662-4586-ae5a-68aa31961d39" (UID: "739f1389-f662-4586-ae5a-68aa31961d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:06.788009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.787985 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1389-f662-4586-ae5a-68aa31961d39-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:06.791134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.791111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5" pod "739f1389-f662-4586-ae5a-68aa31961d39" (UID: "739f1389-f662-4586-ae5a-68aa31961d39"). InnerVolumeSpecName "kube-api-access-fdql5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:06.801446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.801418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials" pod "739f1389-f662-4586-ae5a-68aa31961d39" (UID: "739f1389-f662-4586-ae5a-68aa31961d39"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:06.804826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.804796 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc" Oct 30 22:25:06.806548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.806527 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:06.806657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.806578 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp"] Oct 30 22:25:06.836338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.830104 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:25:06.836338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.830155 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp"] Oct 30 22:25:06.836338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.830338 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.843349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.843313 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-01dc4dac3e0d\"" Oct 30 22:25:06.845017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.844694 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d5x8z\"" Oct 30 22:25:06.860738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.860716 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp"] Oct 30 22:25:06.873150 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.873090 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ll9b7 plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" podUID="48663894-5ae6-4f12-a6fc-0941d162ebb4" Oct 30 22:25:06.889708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.889653 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.889885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.889861 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.890581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.890558 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.890958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.890937 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9b7\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.891199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.891182 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:06.891345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.891318 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdql5\" (UniqueName: \"kubernetes.io/projected/739f1389-f662-4586-ae5a-68aa31961d39-kube-api-access-fdql5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:06.966861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.966827 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc"] Oct 30 22:25:06.972672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.972645 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-hb8sc"] Oct 30 22:25:06.976863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.976839 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:25:06.992604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.992561 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.992779 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.992736 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-01dc4dac3e0d: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:06.992779 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.992762 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:06.992910 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.992831 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:07.492811715 +0000 UTC m=+2731.442766651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:06.992910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.992900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.993041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.993007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9b7\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.993205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.993055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.993205 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.993055 5036 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:06.993205 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.993179 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:07.493136923 +0000 UTC m=+2731.443091865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:06.994196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:06.994171 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:06.996201 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.996181 5036 projected.go:196] Error preparing data for projected volume kube-api-access-ll9b7 for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:06.996339 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:06.996254 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7 podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:07.496237917 +0000 UTC m=+2731.446192854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9b7" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:07.094397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.094363 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\") pod \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " Oct 30 22:25:07.094564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.094406 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\") pod \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " Oct 30 22:25:07.094612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.094571 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\") pod \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " Oct 30 22:25:07.094824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.094806 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\") pod \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\" (UID: \"62ac6157-31df-48a0-80c1-edaa6e4d59d0\") " Oct 30 22:25:07.094980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.094921 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume" pod "62ac6157-31df-48a0-80c1-edaa6e4d59d0" (UID: "62ac6157-31df-48a0-80c1-edaa6e4d59d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:07.095449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.095420 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume" pod "62ac6157-31df-48a0-80c1-edaa6e4d59d0" (UID: "62ac6157-31df-48a0-80c1-edaa6e4d59d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:07.095724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.095704 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:07.095785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.095734 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ac6157-31df-48a0-80c1-edaa6e4d59d0-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:07.096675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.096655 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials" pod "62ac6157-31df-48a0-80c1-edaa6e4d59d0" (UID: "62ac6157-31df-48a0-80c1-edaa6e4d59d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:07.096805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.096792 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d" pod "62ac6157-31df-48a0-80c1-edaa6e4d59d0" (UID: "62ac6157-31df-48a0-80c1-edaa6e4d59d0"). InnerVolumeSpecName "kube-api-access-29r5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:07.204781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.204658 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:07.219400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.219321 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r5d\" (UniqueName: \"kubernetes.io/projected/62ac6157-31df-48a0-80c1-edaa6e4d59d0-kube-api-access-29r5d\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:07.528789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.528697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:07.529315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.528836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9b7\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:07.529315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.528964 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:07.529463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.529317 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-01dc4dac3e0d: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:07.529463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.529340 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:07.529463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.529404 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:08.529382754 +0000 UTC m=+2732.479337696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:07.529818 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.529797 5036 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:07.529887 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.529859 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:08.529843507 +0000 UTC m=+2732.479798445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:07.535533 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.535505 5036 projected.go:196] Error preparing data for projected volume kube-api-access-ll9b7 for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:07.535703 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.535573 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7 podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:08.535556615 +0000 UTC m=+2732.485511541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9b7" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:07.745163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745129 5036 generic.go:356] "Generic (PLEG): container finished" podID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" containerID="83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356" exitCode=0 Oct 30 22:25:07.745348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745187 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" Oct 30 22:25:07.745348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745222 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" event={"ID":"62ac6157-31df-48a0-80c1-edaa6e4d59d0","Type":"ContainerDied","Data":"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356"} Oct 30 22:25:07.745348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745296 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx" event={"ID":"62ac6157-31df-48a0-80c1-edaa6e4d59d0","Type":"ContainerDied","Data":"5012b5337e8319b1a4ca3a4f482fda3978d51df69def259ba1c0d774921cd095"} Oct 30 22:25:07.745348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745324 5036 scope.go:122] "RemoveContainer" containerID="83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356" Oct 30 22:25:07.745763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.745744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:07.756951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.756923 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:07.759733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.759712 5036 scope.go:122] "RemoveContainer" containerID="83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356" Oct 30 22:25:07.760054 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:07.760032 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356\": container with ID starting with 83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356 not found: ID does not exist" containerID="83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356" Oct 30 22:25:07.760127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.760066 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356"} err="failed to get container status \"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356\": rpc error: code = NotFound desc = could not find container \"83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356\": container with ID starting with 83d1eab4d46ccd43338a8591f88dbe23c38254e40b405c85686d0969930db356 not found: ID does not exist" Oct 30 22:25:07.793424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.793338 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:25:07.796567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.796544 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7698d5fd68-bzqjx"] Oct 30 22:25:07.834547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.834514 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\") pod \"48663894-5ae6-4f12-a6fc-0941d162ebb4\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " Oct 30 22:25:07.834956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.834932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume" pod "48663894-5ae6-4f12-a6fc-0941d162ebb4" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:07.835603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:07.835580 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:08.124069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.123978 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj"] Oct 30 22:25:08.124916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.124896 5036 cpu_manager.go:414] "containerMap: removing container" podUID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" containerName="payload-processing" Oct 30 22:25:08.124916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.124917 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" containerName="payload-processing" Oct 30 22:25:08.125092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.125041 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" containerName="payload-processing" Oct 30 22:25:08.130195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.130173 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.139033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.138365 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj"] Oct 30 22:25:08.139033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.139022 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:08.139246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.139143 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:25:08.139246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.139210 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:08.139381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.139305 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:08.191656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.191624 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ac6157-31df-48a0-80c1-edaa6e4d59d0" path="/var/lib/kubelet/pods/62ac6157-31df-48a0-80c1-edaa6e4d59d0/volumes" Oct 30 22:25:08.192087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.192072 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739f1389-f662-4586-ae5a-68aa31961d39" path="/var/lib/kubelet/pods/739f1389-f662-4586-ae5a-68aa31961d39/volumes" Oct 30 22:25:08.265787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.265753 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.266079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.266058 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.266189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.266108 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jw9f\" (UniqueName: \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.367937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.367900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.368116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.368083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.368116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.368109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jw9f\" (UniqueName: \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.368591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.368567 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.370656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.370636 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.381134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.381079 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jw9f\" (UniqueName: \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.457477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.457441 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:08.572049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.571658 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9b7\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.572155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.572404 5036 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-01dc4dac3e0d: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.572425 5036 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.572490 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:10.572471092 +0000 UTC m=+2734.522426042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : secret "agc-cred-01dc4dac3e0d" not found Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.572507 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume\") pod \"payload-processing-external-model-768f4bb6dd-shnbp\" (UID: \"48663894-5ae6-4f12-a6fc-0941d162ebb4\") " pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.572620 5036 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:08.572831 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.572660 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:10.572648335 +0000 UTC m=+2734.522603277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : configmap "payload-processing-external-model-plugins" not found Oct 30 22:25:08.575063 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.575045 5036 projected.go:196] Error preparing data for projected volume kube-api-access-ll9b7 for pod models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:08.575146 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:08.575098 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7 podName:48663894-5ae6-4f12-a6fc-0941d162ebb4 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:10.57508723 +0000 UTC m=+2734.525042154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9b7" (UniqueName: "kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7") pod "payload-processing-external-model-768f4bb6dd-shnbp" (UID: "48663894-5ae6-4f12-a6fc-0941d162ebb4") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 22:25:08.758351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.758316 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp" Oct 30 22:25:08.771121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.771004 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj"] Oct 30 22:25:08.772164 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:25:08.772133 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c3bfce_e426_49c9_abb9_d3d6b85cda8c.slice/crio-7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a WatchSource:0}: Error finding container 7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a: Status 404 returned error can't find the container with id 7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a Oct 30 22:25:08.829359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.829315 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp"] Oct 30 22:25:08.841968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.841937 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-768f4bb6dd-shnbp"] Oct 30 22:25:08.898536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.898497 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:25:08.905091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.905067 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:08.931082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.913421 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-28d39b-dockercfg-k4nbw\"" Oct 30 22:25:08.931082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.913498 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-28d39b\"" Oct 30 22:25:08.931082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.915810 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:25:08.944698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.944606 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:25:08.957935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.957898 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:08.969731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.969693 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:25:08.980323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.980186 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48663894-5ae6-4f12-a6fc-0941d162ebb4-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:08.980529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.980346 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9b7\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-kube-api-access-ll9b7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:08.980529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:08.980363 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48663894-5ae6-4f12-a6fc-0941d162ebb4-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:09.097343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.083883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4h2\" (UniqueName: \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.097343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.083998 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.097343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.084038 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqzj\" (UniqueName: \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.097343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.084079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.224321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.202319 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4h2\" (UniqueName: \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.224321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.202428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.224321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.202460 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqzj\" (UniqueName: \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.224321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.202494 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.272331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.225763 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.272331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.236536 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.303315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.297874 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqzj\" (UniqueName: \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\") pod \"payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.303315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.298490 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4h2\" (UniqueName: \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\") pod \"payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.563106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.554710 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:09.588289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.583309 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:09.771411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.771312 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" event={"ID":"93c3bfce-e426-49c9-abb9-d3d6b85cda8c","Type":"ContainerStarted","Data":"91e42016320b7b55ef060bcf74703f2873c702e2432385254c48f6d77dab48e8"} Oct 30 22:25:09.771411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.771384 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" event={"ID":"93c3bfce-e426-49c9-abb9-d3d6b85cda8c","Type":"ContainerStarted","Data":"7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a"} Oct 30 22:25:09.791875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:09.791812 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" podStartSLOduration=1.7917935200000001 podStartE2EDuration="1.79179352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:09.787933024 +0000 UTC m=+2733.737887973" watchObservedRunningTime="2026-10-30 22:25:09.79179352 +0000 UTC m=+2733.741748467" Oct 30 22:25:10.051202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.044580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:25:10.051202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.050232 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:25:10.054716 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:25:10.052210 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c05ac92_b3ae_4da1_b68c_292efe2a3fbb.slice/crio-f87041fad1e30e4e327cb45597a8202a94a1d0c776260d32a86c31a8783e4d9a WatchSource:0}: Error finding container f87041fad1e30e4e327cb45597a8202a94a1d0c776260d32a86c31a8783e4d9a: Status 404 returned error can't find the container with id f87041fad1e30e4e327cb45597a8202a94a1d0c776260d32a86c31a8783e4d9a Oct 30 22:25:10.192708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.192681 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48663894-5ae6-4f12-a6fc-0941d162ebb4" path="/var/lib/kubelet/pods/48663894-5ae6-4f12-a6fc-0941d162ebb4/volumes" Oct 30 22:25:10.782192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.782155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" event={"ID":"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb","Type":"ContainerStarted","Data":"5d7f89d61f808214f9f84138cb069a069bcd1c52d26ffcab25d0093d2f98678a"} Oct 30 22:25:10.782659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.782214 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" event={"ID":"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb","Type":"ContainerStarted","Data":"f87041fad1e30e4e327cb45597a8202a94a1d0c776260d32a86c31a8783e4d9a"} Oct 30 22:25:10.782659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.782249 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:10.784042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.784016 5036 generic.go:356] "Generic (PLEG): container finished" podID="93c3bfce-e426-49c9-abb9-d3d6b85cda8c" containerID="91e42016320b7b55ef060bcf74703f2873c702e2432385254c48f6d77dab48e8" exitCode=0 Oct 30 22:25:10.784165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.784092 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" event={"ID":"93c3bfce-e426-49c9-abb9-d3d6b85cda8c","Type":"ContainerDied","Data":"91e42016320b7b55ef060bcf74703f2873c702e2432385254c48f6d77dab48e8"} Oct 30 22:25:10.785918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.785893 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" event={"ID":"cfedfbaa-c55d-4483-84db-e74e60da29ff","Type":"ContainerStarted","Data":"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7"} Oct 30 22:25:10.785987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.785932 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" event={"ID":"cfedfbaa-c55d-4483-84db-e74e60da29ff","Type":"ContainerStarted","Data":"386d0673b47228e0c7e58c4c55508333bddc71168edfe860319c2885ff98b2be"} Oct 30 22:25:10.785987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.785968 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:10.802453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.802404 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" podStartSLOduration=2.80238843 podStartE2EDuration="2.80238843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:10.800176783 +0000 UTC m=+2734.750131728" watchObservedRunningTime="2026-10-30 22:25:10.80238843 +0000 UTC m=+2734.752343383" Oct 30 22:25:10.842138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:10.842078 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" podStartSLOduration=2.842059773 podStartE2EDuration="2.842059773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:10.838525953 +0000 UTC m=+2734.788480900" watchObservedRunningTime="2026-10-30 22:25:10.842059773 +0000 UTC m=+2734.792014721" Oct 30 22:25:12.056468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.056394 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:12.226848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.226815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\") pod \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " Oct 30 22:25:12.227050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.226900 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\") pod \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " Oct 30 22:25:12.227130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.227108 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\" (UniqueName: \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\") pod \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\" (UID: \"93c3bfce-e426-49c9-abb9-d3d6b85cda8c\") " Oct 30 22:25:12.227398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.227365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca" pod "93c3bfce-e426-49c9-abb9-d3d6b85cda8c" (UID: "93c3bfce-e426-49c9-abb9-d3d6b85cda8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:12.228040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.228010 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:12.229461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.229429 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca" pod "93c3bfce-e426-49c9-abb9-d3d6b85cda8c" (UID: "93c3bfce-e426-49c9-abb9-d3d6b85cda8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:12.229558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.229491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f" pod "93c3bfce-e426-49c9-abb9-d3d6b85cda8c" (UID: "93c3bfce-e426-49c9-abb9-d3d6b85cda8c"). InnerVolumeSpecName "kube-api-access-8jw9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:12.330608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.330521 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:12.330608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.330563 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jw9f\" (UniqueName: \"kubernetes.io/projected/93c3bfce-e426-49c9-abb9-d3d6b85cda8c-kube-api-access-8jw9f\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:12.811628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.811596 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" Oct 30 22:25:12.811784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.811623 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj" event={"ID":"93c3bfce-e426-49c9-abb9-d3d6b85cda8c","Type":"ContainerDied","Data":"7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a"} Oct 30 22:25:12.811784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:12.811684 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee43f1fd700966d4d029c66198cb46a81fb22a109ef22f85e6e0e5c4179009a" Oct 30 22:25:17.887141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:17.887025 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:18.179347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.147362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:25:18.179347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.164691 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:18.593340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.588904 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:25:18.653436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.653387 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:18.752076 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:18.752035 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:25:18.817432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.794587 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dl8xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:25:18.817432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:18.794681 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dl8xl" podUID="bf00519b-3c51-4867-bd8a-b68854743ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:19.227292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:19.220388 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:21.798924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:21.798896 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:25:21.799365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:21.798973 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:25:33.269845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.269804 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:25:33.271178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.271152 5036 cpu_manager.go:414] "containerMap: removing container" podUID="93c3bfce-e426-49c9-abb9-d3d6b85cda8c" containerName="revoke-keys" Oct 30 22:25:33.271178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.271178 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c3bfce-e426-49c9-abb9-d3d6b85cda8c" containerName="revoke-keys" Oct 30 22:25:33.271469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.271450 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c3bfce-e426-49c9-abb9-d3d6b85cda8c" containerName="revoke-keys" Oct 30 22:25:33.275468 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.275449 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.289026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.288999 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4b2bb0-openshift-default-dockercfg-567bk\"" Oct 30 22:25:33.296456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.296419 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:25:33.367434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367402 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367452 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367574 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367617 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367833 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.367977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.367956 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8phz\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.368171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.368149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.475480 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.475553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.475663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8phz\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.475847 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.475900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.476037 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.476697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.478840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.477000 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.489336 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.489447 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.489487 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.489572 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.490734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.493292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.491055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.519316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.506752 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.542339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.536005 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.563676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.562919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.563676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.563602 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8phz\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\") pod \"e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:33.594375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:33.592643 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:34.142180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.142140 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:25:34.149437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.149410 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.162394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.162367 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c92aec-openshift-default-dockercfg-jtz49\"" Oct 30 22:25:34.175130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.175088 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:25:34.179034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.178995 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:34.179163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.179080 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:34.179163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.179123 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:34.207110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.207075 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.283734 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.283848 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.283940 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284012 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284110 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284246 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7bn\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284309 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.286298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.284402 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.394663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.394809 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.394889 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.394970 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.395030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.395114 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.395231 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7bn\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.395287 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.400289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.395312 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.425335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.403732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.425335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.404801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.425335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.405053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.447300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.443388 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.447300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.444219 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.460844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.458580 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.460844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.460783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.476464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.465347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7bn\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.502289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.477958 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.502289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.491581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:34.893302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.893173 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerStarted","Data":"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e"} Oct 30 22:25:34.893302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:34.893226 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerStarted","Data":"11ea1109bf2f53619be420b0c491405e665594d98999b77a3dea4d944c0c46e2"} Oct 30 22:25:35.153298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.146483 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podStartSLOduration=2.146466581 podStartE2EDuration="2.146466581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:34.924494928 +0000 UTC m=+2758.874449873" watchObservedRunningTime="2026-10-30 22:25:35.146466581 +0000 UTC m=+2759.096421526" Oct 30 22:25:35.153298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.147356 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:25:35.153298 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:25:35.150917 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d40a53b_e3fe_49dd_a0a0_9720de857402.slice/crio-0891736f5ba6b458d0aa47463505a82107d4c149d792bcb8fac1962ba53c9694 WatchSource:0}: Error finding container 0891736f5ba6b458d0aa47463505a82107d4c149d792bcb8fac1962ba53c9694: Status 404 returned error can't find the container with id 0891736f5ba6b458d0aa47463505a82107d4c149d792bcb8fac1962ba53c9694 Oct 30 22:25:35.155316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.154532 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:35.155316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.154601 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:35.155316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.154631 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:35.594805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.594763 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:35.602099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.602068 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:35.906024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.905991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerStarted","Data":"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6"} Oct 30 22:25:35.906233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.906057 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerStarted","Data":"0891736f5ba6b458d0aa47463505a82107d4c149d792bcb8fac1962ba53c9694"} Oct 30 22:25:35.906621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.906600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:35.907517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.907492 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:25:35.931753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:35.931713 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podStartSLOduration=1.9317012409999998 podStartE2EDuration="1.931701241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:35.927653606 +0000 UTC m=+2759.877608552" watchObservedRunningTime="2026-10-30 22:25:35.931701241 +0000 UTC m=+2759.881656187" Oct 30 22:25:36.497707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:36.497667 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:36.503471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:36.503448 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:36.918347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:36.918309 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:36.919420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:36.919400 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:25:40.357402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:40.357340 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj"] Oct 30 22:25:40.402294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:40.398819 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f8222-f2xjj"] Oct 30 22:25:42.192112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:42.192081 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c3bfce-e426-49c9-abb9-d3d6b85cda8c" path="/var/lib/kubelet/pods/93c3bfce-e426-49c9-abb9-d3d6b85cda8c/volumes" Oct 30 22:25:47.674300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:47.674243 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:25:47.674791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:47.674518 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" containerID="cri-o://0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460" gracePeriod=30 Oct 30 22:25:48.197897 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:48.197848 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:25:48.206046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.206020 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc"] Oct 30 22:25:48.212770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.212743 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc"] Oct 30 22:25:48.212889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.212846 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.217679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.217656 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:48.217793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.217737 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:48.275362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.275331 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:48.275582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.275566 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:25:48.288700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.288672 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.288835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.288787 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.288878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.288856 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p4f\" (UniqueName: \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.390543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.390508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.390761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.390602 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.390761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.390667 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p4f\" (UniqueName: \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.391222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.391198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.393414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.393394 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.403812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.403775 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p4f\" (UniqueName: \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\") pod \"maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.534208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.534120 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:48.901259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:48.901230 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc"] Oct 30 22:25:48.902334 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:25:48.902306 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8868156c_1bb0_4638_8dfd_9272126af38e.slice/crio-8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e WatchSource:0}: Error finding container 8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e: Status 404 returned error can't find the container with id 8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e Oct 30 22:25:49.156601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:49.156505 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" event={"ID":"8868156c-1bb0-4638-8dfd-9272126af38e","Type":"ContainerStarted","Data":"8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e"} Oct 30 22:25:49.557492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:49.557457 5036 scope.go:122] "RemoveContainer" containerID="0a7e1ee8b06cea5ee854971619353cc0adc66acbda0728b09bce76f4baeee3a4" Oct 30 22:25:50.167447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:50.167409 5036 generic.go:356] "Generic (PLEG): container finished" podID="8868156c-1bb0-4638-8dfd-9272126af38e" containerID="a72a20d55c075feea1f3fbaeeb35a4c73038f1e2e8d6a09320f13238c8abfcbb" exitCode=0 Oct 30 22:25:50.167842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:50.167491 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" event={"ID":"8868156c-1bb0-4638-8dfd-9272126af38e","Type":"ContainerDied","Data":"a72a20d55c075feea1f3fbaeeb35a4c73038f1e2e8d6a09320f13238c8abfcbb"} Oct 30 22:25:51.370248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.370222 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:51.478023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.477929 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\") pod \"8868156c-1bb0-4638-8dfd-9272126af38e\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " Oct 30 22:25:51.478214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.478139 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\") pod \"8868156c-1bb0-4638-8dfd-9272126af38e\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " Oct 30 22:25:51.478329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.478315 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\" (UniqueName: \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\") pod \"8868156c-1bb0-4638-8dfd-9272126af38e\" (UID: \"8868156c-1bb0-4638-8dfd-9272126af38e\") " Oct 30 22:25:51.478538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.478509 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca" pod "8868156c-1bb0-4638-8dfd-9272126af38e" (UID: "8868156c-1bb0-4638-8dfd-9272126af38e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:51.479044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.479009 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:51.480928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.480898 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca" pod "8868156c-1bb0-4638-8dfd-9272126af38e" (UID: "8868156c-1bb0-4638-8dfd-9272126af38e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:51.481042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.480942 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f" pod "8868156c-1bb0-4638-8dfd-9272126af38e" (UID: "8868156c-1bb0-4638-8dfd-9272126af38e"). InnerVolumeSpecName "kube-api-access-h2p4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:51.580390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.580349 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2p4f\" (UniqueName: \"kubernetes.io/projected/8868156c-1bb0-4638-8dfd-9272126af38e-kube-api-access-h2p4f\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:51.580390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:51.580393 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868156c-1bb0-4638-8dfd-9272126af38e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.187935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:52.187902 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" Oct 30 22:25:52.190488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:52.190455 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc" event={"ID":"8868156c-1bb0-4638-8dfd-9272126af38e","Type":"ContainerDied","Data":"8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e"} Oct 30 22:25:52.190627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:52.190495 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7c6e0b76b28b95b46b3d416cc6271ee79c9c42d3a64ab01c902aabe528395e" Oct 30 22:25:52.999470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:52.999447 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:25:53.009130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009105 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009150 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009347 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009369 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009393 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009530 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.009575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009541 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.009750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009727 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009737 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:53.009750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.009988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009782 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.009988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.009860 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\") pod \"bbdcb908-5eb3-4825-91c5-0e3884697059\" (UID: \"bbdcb908-5eb3-4825-91c5-0e3884697059\") " Oct 30 22:25:53.010218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.010796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010768 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.010910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010802 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdcb908-5eb3-4825-91c5-0e3884697059-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.010910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010818 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.010910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010832 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.010910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.010847 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.011914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.011889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "kube-api-access-s97kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:53.012093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.012056 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.012613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.012584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:53.012713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.012633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo" pod "bbdcb908-5eb3-4825-91c5-0e3884697059" (UID: "bbdcb908-5eb3-4825-91c5-0e3884697059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:53.112581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.112548 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.112581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.112575 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.112581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.112587 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdcb908-5eb3-4825-91c5-0e3884697059-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.112810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.112597 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s97kb\" (UniqueName: \"kubernetes.io/projected/bbdcb908-5eb3-4825-91c5-0e3884697059-kube-api-access-s97kb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.198341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.198305 5036 generic.go:356] "Generic (PLEG): container finished" podID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerID="0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460" exitCode=0 Oct 30 22:25:53.198538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.198363 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" Oct 30 22:25:53.198538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.198383 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerDied","Data":"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460"} Oct 30 22:25:53.198538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.198447 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx" event={"ID":"bbdcb908-5eb3-4825-91c5-0e3884697059","Type":"ContainerDied","Data":"5d65f3b805fefd456e1b1db1c32a53d37ebe9707e2775b75401be043884e9b21"} Oct 30 22:25:53.198538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.198472 5036 scope.go:122] "RemoveContainer" containerID="0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460" Oct 30 22:25:53.248689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.248446 5036 scope.go:122] "RemoveContainer" containerID="5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4" Oct 30 22:25:53.289911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.289878 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:25:53.295182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.295150 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6f8222-openshift-default-64774b79d9-kc6fx"] Oct 30 22:25:53.299437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.299417 5036 scope.go:122] "RemoveContainer" containerID="0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460" Oct 30 22:25:53.299756 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:53.299723 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460\": container with ID starting with 0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460 not found: ID does not exist" containerID="0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460" Oct 30 22:25:53.299879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.299763 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460"} err="failed to get container status \"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460\": rpc error: code = NotFound desc = could not find container \"0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460\": container with ID starting with 0c054e7f5525f066b86a39bf3bc735f67c74d19c9a6e70667fa726afeeab4460 not found: ID does not exist" Oct 30 22:25:53.299879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.299784 5036 scope.go:122] "RemoveContainer" containerID="5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4" Oct 30 22:25:53.300113 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:53.300084 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4\": container with ID starting with 5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4 not found: ID does not exist" containerID="5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4" Oct 30 22:25:53.300157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:53.300119 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4"} err="failed to get container status \"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4\": rpc error: code = NotFound desc = could not find container \"5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4\": container with ID starting with 5ee4ff1c67d45948e1c6e1674656c05bd0db9834b72fe0db26e81348903f27f4 not found: ID does not exist" Oct 30 22:25:54.046680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.046641 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:25:54.048894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.048869 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.049422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.049403 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.049634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.049622 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8868156c-1bb0-4638-8dfd-9272126af38e" containerName="revoke-keys" Oct 30 22:25:54.049713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.049704 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8868156c-1bb0-4638-8dfd-9272126af38e" containerName="revoke-keys" Oct 30 22:25:54.049821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.049812 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.049897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.049888 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.050241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.050230 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8868156c-1bb0-4638-8dfd-9272126af38e" containerName="revoke-keys" Oct 30 22:25:54.050379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.050367 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.050464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.050455 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" containerName="istio-proxy" Oct 30 22:25:54.058109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.058070 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.076085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.076031 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:25:54.076534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.076503 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-58d5fa-openshift-default-dockercfg-tdq82\"" Oct 30 22:25:54.179561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.179737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179571 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px944\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.179737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179600 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.179880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179775 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.179945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.179945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.179916 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.180058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.180006 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.180058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.180031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.180179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.180106 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.190584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.190543 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdcb908-5eb3-4825-91c5-0e3884697059" path="/var/lib/kubelet/pods/bbdcb908-5eb3-4825-91c5-0e3884697059/volumes" Oct 30 22:25:54.287606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.287566 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.289045 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.289018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.289758 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.289737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px944\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.289941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.289924 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.290177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.290161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.290386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.290369 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.297414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.297357 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.297932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.297909 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.298090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.298074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.298333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.288780 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.298523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.291050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.310285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.294788 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.311128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.298568 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.311319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.298788 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.312804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.312784 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.331289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.326221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px944\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.331749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.331727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.332321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.332259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\") pod \"e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:54.395357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:54.395324 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:55.073047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.069864 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:25:55.080244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.080192 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:55.080474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.080310 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:55.080474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.080361 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:55.445294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.439934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerStarted","Data":"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7"} Oct 30 22:25:55.445294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:55.440023 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerStarted","Data":"1aefa4149e7208e256a51362b74741d859e8323c09d6634351f70f450457f4f6"} Oct 30 22:25:56.359291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:56.358367 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:56.679864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:56.679794 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podStartSLOduration=2.6797686819999997 podStartE2EDuration="2.679768682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.632856157 +0000 UTC m=+2780.582811114" watchObservedRunningTime="2026-10-30 22:25:56.679768682 +0000 UTC m=+2780.629723629" Oct 30 22:25:56.783326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:56.782845 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:56.828681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:56.828629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:25:56.902292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:56.900399 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:57.064470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.064316 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd"] Oct 30 22:25:57.173288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.157305 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504c1f76-9k9qd"] Oct 30 22:25:57.363352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.362360 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:57.371292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.364950 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:25:57.414289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.411399 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:57.414289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.411681 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:25:57.757517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.757480 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:57.855191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.848939 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:57.855191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.849534 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:25:57.921316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.919444 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:25:57.921316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.919905 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:25:57.963309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:57.944610 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:58.037353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.030783 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:25:58.184314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.170570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:25:58.288660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.189902 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:58.429310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.418488 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:58.429310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.418943 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:58.801394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.766940 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:25:58.858289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.852459 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:25:58.906335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.903778 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-5bm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:25:58.906335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:58.903840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-5bm7l" podUID="b9b6de7a-8d31-4f0e-990b-81525b0912d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:59.049393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.034594 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mm6v2" podUID="0b9abb0b-7989-467e-86a9-41bbe7ce3a59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:59.202317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.196809 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89e1157-560b-4c02-9396-0ac8b9ecc02a" path="/var/lib/kubelet/pods/f89e1157-560b-4c02-9396-0ac8b9ecc02a/volumes" Oct 30 22:25:59.202317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.198579 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:59.417293 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:25:59.261951 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.077s" Oct 30 22:25:59.514350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.513868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:25:59.514350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.514125 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:25:59.542430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.432580 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:25:59.554003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.432727 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:25:59.736389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.649730 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:59.746423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.650186 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:25:59.883348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.881518 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:25:59.883348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.881606 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:26:00.090766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.956126 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:00.090766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:25:59.956529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:00.090766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.062756 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:26:00.149347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.091370 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:26:00.149823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.149770 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:00.353305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.347120 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 30 22:26:00.582387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.581755 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:26:00.863968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.863871 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:00.922298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.863871 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:01.029871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.863902 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:01.056293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:00.922601 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 30 22:26:01.056293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.048586 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:01.190748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.190707 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:01.223307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.222580 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:01.223307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.223065 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:26:01.634803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.601458 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:01.886356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:01.873298 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:02.024430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.024376 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:02.030378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.030332 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:02.052233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.052185 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:02.192349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.191770 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:26:02.225623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.225579 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:02.604065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.603511 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:02.890641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:02.890596 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:26:03.036168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.036125 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:26:03.037822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.036479 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:03.067783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.067739 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:03.128293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.124089 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:26:03.206088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.205415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:03.242345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.229415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:26:03.605877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:03.605780 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:26:04.043416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.043362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:04.097327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.083936 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:04.115289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.087015 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:04.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.267354 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:04.321957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.291250 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:26:04.609358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.607366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:04.926914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:04.900989 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:05.088296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.062576 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:05.118602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.107219 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:05.175455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.175398 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:05.270967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.269713 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:26:05.324461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.323376 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:05.466299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.463597 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:26:05.755509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:05.742899 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:26:06.103615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.103506 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:06.173135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.141610 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:06.339659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.339595 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:06.341779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.341729 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:06.751148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.747919 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:26:06.809418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.809368 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:06.873990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.873954 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:26:06.923847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:06.923814 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:26:07.014349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:07.010324 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:26:09.052908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:09.052871 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl"] Oct 30 22:26:09.061772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:09.061725 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b601d01b-22dhl"] Oct 30 22:26:10.221891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:10.221817 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4865a7-34c4-4a01-bebd-cb33f29e9a16" path="/var/lib/kubelet/pods/2c4865a7-34c4-4a01-bebd-cb33f29e9a16/volumes" Oct 30 22:26:11.458797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:11.457697 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc"] Oct 30 22:26:11.507064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:11.504008 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d8c835-j2nfc"] Oct 30 22:26:12.197147 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:12.197107 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8868156c-1bb0-4638-8dfd-9272126af38e" path="/var/lib/kubelet/pods/8868156c-1bb0-4638-8dfd-9272126af38e/volumes" Oct 30 22:26:12.832149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:12.832108 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:26:12.876575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:12.876540 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:12.876697 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007141 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007347 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007460 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007515 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007606 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007802 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.007834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxlk\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.109891 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxlk\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.109941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110001 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110071 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110143 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110250 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110441 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110564 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110937 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.110994 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.111431 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.113512 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.113603 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.144472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.162768 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxlk\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.228003 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.581975 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:26:13.581973 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ee1dcb_902e_45fc_957f_c3069627d439.slice/crio-09ca63e1889325f74fb6ffaf34a6481043d6d6c821f506a653e65c40f156d61f WatchSource:0}: Error finding container 09ca63e1889325f74fb6ffaf34a6481043d6d6c821f506a653e65c40f156d61f: Status 404 returned error can't find the container with id 09ca63e1889325f74fb6ffaf34a6481043d6d6c821f506a653e65c40f156d61f Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.585949 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.586036 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:13.615467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:13.586082 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.566349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.563721 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" event={"ID":"43ee1dcb-902e-45fc-957f-c3069627d439","Type":"ContainerStarted","Data":"4685d4b9acb233f3312f6023502ac733a528c01ab7006ee14cfcdd7fdb31a9d6"} Oct 30 22:26:14.566349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.563792 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" event={"ID":"43ee1dcb-902e-45fc-957f-c3069627d439","Type":"ContainerStarted","Data":"09ca63e1889325f74fb6ffaf34a6481043d6d6c821f506a653e65c40f156d61f"} Oct 30 22:26:14.568848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.568820 5036 generic.go:356] "Generic (PLEG): container finished" podID="f98fec20-adfa-4090-a5ed-39d187302155" containerID="007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95" exitCode=0 Oct 30 22:26:14.569108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.568908 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerDied","Data":"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95"} Oct 30 22:26:14.570540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.570404 5036 scope.go:122] "RemoveContainer" containerID="007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95" Oct 30 22:26:14.573432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573022 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.573432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573097 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.573432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573134 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.573432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573216 5036 generic.go:356] "Generic (PLEG): container finished" podID="71ecec29-954d-4306-98fe-62d3e2085f81" containerID="977b69c03187d6938e9460990fc7d13b2bbdefa1d64d63622c7882324b8cc88b" exitCode=0 Oct 30 22:26:14.573432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573325 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerDied","Data":"977b69c03187d6938e9460990fc7d13b2bbdefa1d64d63622c7882324b8cc88b"} Oct 30 22:26:14.574406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.573912 5036 scope.go:122] "RemoveContainer" containerID="977b69c03187d6938e9460990fc7d13b2bbdefa1d64d63622c7882324b8cc88b" Oct 30 22:26:14.583942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.583906 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.584055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.583996 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.584055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.584041 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.596228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.596161 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podStartSLOduration=2.596142223 podStartE2EDuration="2.596142223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:14.594024776 +0000 UTC m=+2798.543979750" watchObservedRunningTime="2026-10-30 22:26:14.596142223 +0000 UTC m=+2798.546097170" Oct 30 22:26:14.603406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.603375 5036 generic.go:356] "Generic (PLEG): container finished" podID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerID="a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354" exitCode=0 Oct 30 22:26:14.603653 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.603629 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerDied","Data":"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354"} Oct 30 22:26:14.609449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.609422 5036 scope.go:122] "RemoveContainer" containerID="a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354" Oct 30 22:26:14.635184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.635133 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.645092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.644974 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:14.645590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:14.645545 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:15.232074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.232042 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:15.233049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.233024 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:15.233257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.233225 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:15.276483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.276374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:26:15.669596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.667584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerStarted","Data":"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9"} Oct 30 22:26:15.674550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.674494 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerStarted","Data":"cecdfb7226fca9d79144767e66f0812ea60298c335980a91e809478a4fb4792c"} Oct 30 22:26:15.677511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.677478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerStarted","Data":"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386"} Oct 30 22:26:15.988348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.987924 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:26:15.988348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.988127 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:15.988348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:15.988195 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:16.236295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.231927 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.236295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.231986 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:16.358353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.357397 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:26:16.392289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.391728 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:26:16.392289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.391853 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.392289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.391896 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:16.590891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.590854 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:26:16.591999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.591184 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.592202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.592172 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:17.012294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.995902 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.012294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:16.995960 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:17.054862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.052494 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:17.096370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.088375 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:17.279548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.279443 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.279548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.279530 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:17.397296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.390247 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.397296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.390356 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:17.580367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.575667 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.580367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.575756 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:17.607312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.599419 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:17.912293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.911458 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:17.990301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.988566 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.990301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:17.988644 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:18.053595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.053544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:18.095687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.095640 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:26:18.201352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.097136 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:18.201352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.185385 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:26:18.265746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.265703 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.372459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.372391 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:18.398294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.347239 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:26:18.519025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.518231 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:18.524592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.523337 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.524592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.523391 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:18.648291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.602219 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:18.648924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.643392 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.649151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.649112 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:18.675559 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:26:18.645503 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:26:18.766537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.766501 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:26:18.949292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.941108 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:18.949292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.941709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:26:18.978406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.978369 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.978695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:18.978666 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:19.117205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.117159 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.191772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.191716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:19.224660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.224550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.239000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.238967 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.243037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.242986 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:19.247643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.247403 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.409843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.395793 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.409843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.395853 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:19.622838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.622748 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.623098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.623068 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:19.623678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.623648 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.651529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.651496 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:26:19.668357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.654120 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:26:19.819310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.785353 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.954211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.954158 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.995295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.973616 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.995295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:19.973684 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:20.166304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.138497 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:26:20.166304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.138765 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" containerID="cri-o://8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace" gracePeriod=30 Oct 30 22:26:20.554811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.553290 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 30 22:26:20.556782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.555373 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:20.590687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.590637 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:26:20.604093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.604041 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:20.608178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.608125 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 22:26:20.774506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.774450 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:20.774857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.774825 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:20.812568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.812458 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:20.846308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.841955 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:20.846308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.842030 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:20.865291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.852104 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:26:20.866811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.856668 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:20.867032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.867004 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:20.939568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.935760 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 30 22:26:21.002292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:20.991319 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:21.226545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.226459 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:21.226545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.226538 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:21.252433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.252397 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:21.252645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.252461 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:21.418742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.418705 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:21.436289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.418980 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:21.492346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.453996 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:26:21.601288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.597189 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv"] Oct 30 22:26:21.619293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.607473 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:21.684335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.672650 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:21.684335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.641381 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:21.684335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.673391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:21.684335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.673620 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:21.916293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.915348 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:21.940293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.933446 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.983528 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv"] Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.983728 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.988935 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.988988 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:21.983653 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.015022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.013252 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:26:22.135343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.133456 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:26:22.135343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.133519 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:22.135343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.133594 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:22.135343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.133706 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:22.257645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.257546 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.258247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.258205 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 22:26:22.418806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.418768 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.419108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.419071 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:22.474761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.474719 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:22.528124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.528027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.528576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.528550 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82jn\" (UniqueName: \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.528903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.528885 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.578392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.578357 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.588284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.588208 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:22.631890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.630667 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82jn\" (UniqueName: \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.632385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.632361 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.632788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.632769 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.644492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.644465 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.682884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.682435 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 22:26:22.684319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.682477 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" podUID="015c7202-4d5d-461c-99d1-cc3eca48e7b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 22:26:22.698291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.682497 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:22.740033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.739991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82jn\" (UniqueName: \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.779350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.743662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.826338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.822327 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:22.889916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.888581 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:26:22.956297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.955540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:26:22.962406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.960242 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:22.979460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.977709 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.979460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:22.977782 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:23.100008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.099896 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:23.133621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.132536 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm" Oct 30 22:26:23.202588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.202554 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:26:23.249372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.249338 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:26:23.499622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.498098 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:23.499622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.498163 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:23.588933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.585582 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:23.591652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.591626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:23.662334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.607897 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:26:23.662334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.610163 5036 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Oct 30 22:26:23.662334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.610214 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Oct 30 22:26:24.000305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.988354 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:24.000305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:23.988431 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:24.028316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.012915 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Oct 30 22:26:24.072119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.070130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:26:24.130806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.106965 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:24.266166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.266052 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 22:26:24.434321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.415685 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 30 22:26:24.434321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.415779 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 30 22:26:24.484733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.484700 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:26:24.950451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.921972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:26:24.966817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.966749 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:26:24.988576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.984393 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:24.988576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:24.984462 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 30 22:26:25.159097 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:26:25.158129 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb51a57_8b6e_4baf_a14d_5318390192dd.slice/crio-b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125 WatchSource:0}: Error finding container b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125: Status 404 returned error can't find the container with id b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125 Oct 30 22:26:25.169875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.169117 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv"] Oct 30 22:26:25.683298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.670216 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:26:25.715359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.684347 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:26:25.839136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.839109 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:26:25.947052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.947010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" event={"ID":"0fb51a57-8b6e-4baf-a14d-5318390192dd","Type":"ContainerStarted","Data":"19efd60663b57bb22a63d39b9d2f7efe9bba6f7a4ad439a6cb88348e311be77a"} Oct 30 22:26:25.947250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.947082 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" event={"ID":"0fb51a57-8b6e-4baf-a14d-5318390192dd","Type":"ContainerStarted","Data":"b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125"} Oct 30 22:26:25.951665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.951635 5036 generic.go:356] "Generic (PLEG): container finished" podID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerID="8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace" exitCode=0 Oct 30 22:26:25.952278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.951702 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" Oct 30 22:26:25.952455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.951730 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerDied","Data":"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace"} Oct 30 22:26:25.952455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.952406 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n" event={"ID":"4fad8555-d2ee-49ec-a2a7-142f5ab22c10","Type":"ContainerDied","Data":"ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb"} Oct 30 22:26:25.952455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.952436 5036 scope.go:122] "RemoveContainer" containerID="8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace" Oct 30 22:26:25.969792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.969747 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" podStartSLOduration=5.969733024 podStartE2EDuration="5.969733024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:25.964987335 +0000 UTC m=+2809.914942505" watchObservedRunningTime="2026-10-30 22:26:25.969733024 +0000 UTC m=+2809.919687981" Oct 30 22:26:25.970594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.970573 5036 scope.go:122] "RemoveContainer" containerID="12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b" Oct 30 22:26:25.978648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.978626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:26:25.978988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.978964 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:26:25.980231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.980199 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:26:25.986019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.986004 5036 scope.go:122] "RemoveContainer" containerID="8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace" Oct 30 22:26:25.986365 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:26:25.986340 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace\": container with ID starting with 8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace not found: ID does not exist" containerID="8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace" Oct 30 22:26:25.986438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.986371 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace"} err="failed to get container status \"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace\": rpc error: code = NotFound desc = could not find container \"8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace\": container with ID starting with 8a17b7678e3cccb1b1feb03e87b452a2807ec14fc9ce48c7e4db4ffb49993ace not found: ID does not exist" Oct 30 22:26:25.986438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.986389 5036 scope.go:122] "RemoveContainer" containerID="12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b" Oct 30 22:26:25.986649 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:26:25.986629 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b\": container with ID starting with 12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b not found: ID does not exist" containerID="12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b" Oct 30 22:26:25.986950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.986656 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b"} err="failed to get container status \"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b\": rpc error: code = NotFound desc = could not find container \"12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b\": container with ID starting with 12ab25e5b6ff66eae0316d7f6d294320e89404c6621cf51a3d11dd6c4346aa8b not found: ID does not exist" Oct 30 22:26:25.992110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992059 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992146 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992184 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.992595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992504 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992547 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992603 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:25.992795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.992795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992787 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992859 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\") pod \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\" (UID: \"4fad8555-d2ee-49ec-a2a7-142f5ab22c10\") " Oct 30 22:26:25.992982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.992945 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.993214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.993662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993642 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.993753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993666 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.993753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993675 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.993753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993685 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.993753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.993698 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.994626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.994606 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:25.995725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.995697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.995823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.995730 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "kube-api-access-878bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:25.996388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:25.996364 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo" pod "4fad8555-d2ee-49ec-a2a7-142f5ab22c10" (UID: "4fad8555-d2ee-49ec-a2a7-142f5ab22c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:26.096520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.096488 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.096665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.096567 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.096665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.096580 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878bz\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-kube-api-access-878bz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.096665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.096591 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad8555-d2ee-49ec-a2a7-142f5ab22c10-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.276943 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:26:26.276859 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fad8555_d2ee_49ec_a2a7_142f5ab22c10.slice/crio-ca8577fb278097507bf8c446c1d7469cda4fc44efc59e8e95e64793be2fa59bb\": RecentStats: unable to find data in memory cache]" Oct 30 22:26:26.277103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.277018 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:26:26.283172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.283136 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d8c835-openshift-default-5b4666c5b5-g8w2n"] Oct 30 22:26:26.967626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.967587 5036 generic.go:356] "Generic (PLEG): container finished" podID="0fb51a57-8b6e-4baf-a14d-5318390192dd" containerID="19efd60663b57bb22a63d39b9d2f7efe9bba6f7a4ad439a6cb88348e311be77a" exitCode=0 Oct 30 22:26:26.968115 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:26.967673 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" event={"ID":"0fb51a57-8b6e-4baf-a14d-5318390192dd","Type":"ContainerDied","Data":"19efd60663b57bb22a63d39b9d2f7efe9bba6f7a4ad439a6cb88348e311be77a"} Oct 30 22:26:28.228686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.228644 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" path="/var/lib/kubelet/pods/4fad8555-d2ee-49ec-a2a7-142f5ab22c10/volumes" Oct 30 22:26:28.329080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.329050 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:28.386418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.386390 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\" (UniqueName: \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\") pod \"0fb51a57-8b6e-4baf-a14d-5318390192dd\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " Oct 30 22:26:28.386808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.386781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\") pod \"0fb51a57-8b6e-4baf-a14d-5318390192dd\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " Oct 30 22:26:28.386934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.386906 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\") pod \"0fb51a57-8b6e-4baf-a14d-5318390192dd\" (UID: \"0fb51a57-8b6e-4baf-a14d-5318390192dd\") " Oct 30 22:26:28.387379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.387327 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca" pod "0fb51a57-8b6e-4baf-a14d-5318390192dd" (UID: "0fb51a57-8b6e-4baf-a14d-5318390192dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:28.388202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.388173 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.389775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.389743 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca" pod "0fb51a57-8b6e-4baf-a14d-5318390192dd" (UID: "0fb51a57-8b6e-4baf-a14d-5318390192dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:28.389873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.389794 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn" pod "0fb51a57-8b6e-4baf-a14d-5318390192dd" (UID: "0fb51a57-8b6e-4baf-a14d-5318390192dd"). InnerVolumeSpecName "kube-api-access-r82jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:28.489861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.489820 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb51a57-8b6e-4baf-a14d-5318390192dd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.489861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:28.489853 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82jn\" (UniqueName: \"kubernetes.io/projected/0fb51a57-8b6e-4baf-a14d-5318390192dd-kube-api-access-r82jn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.035911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:29.035873 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" event={"ID":"0fb51a57-8b6e-4baf-a14d-5318390192dd","Type":"ContainerDied","Data":"b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125"} Oct 30 22:26:29.036134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:29.035917 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv" Oct 30 22:26:29.036134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:29.035937 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b3f910f553b1da5f27f9bb7113b7460e3640b60ce55fba89314d1e92c62125" Oct 30 22:26:30.303410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:30.303368 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:26:31.869292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:31.868326 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv"] Oct 30 22:26:31.884289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:31.880133 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5313f0-4wflv"] Oct 30 22:26:32.191025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:32.190984 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb51a57-8b6e-4baf-a14d-5318390192dd" path="/var/lib/kubelet/pods/0fb51a57-8b6e-4baf-a14d-5318390192dd/volumes" Oct 30 22:26:35.311052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:35.307165 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" containerID="cri-o://eb99206f09bd7f327938fb280d898de79d7d3121fd64e2b9480b327d2df55090" gracePeriod=25 Oct 30 22:26:36.195607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:36.194756 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:39.260869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.260827 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:26:39.261430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.261025 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" containerID="cri-o://33435b0726978b771285876da812adc3323c198c02cdf42a20ace5adc2fe53fa" gracePeriod=30 Oct 30 22:26:39.809616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.809578 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6"] Oct 30 22:26:39.810612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810586 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.810612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810610 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.810802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810630 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.810802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810638 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.810802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810683 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0fb51a57-8b6e-4baf-a14d-5318390192dd" containerName="revoke-keys" Oct 30 22:26:39.810802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810691 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb51a57-8b6e-4baf-a14d-5318390192dd" containerName="revoke-keys" Oct 30 22:26:39.810944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810921 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.810944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810932 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fad8555-d2ee-49ec-a2a7-142f5ab22c10" containerName="istio-proxy" Oct 30 22:26:39.811009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.810950 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb51a57-8b6e-4baf-a14d-5318390192dd" containerName="revoke-keys" Oct 30 22:26:39.815885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.815856 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:39.822109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.821108 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:39.822109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.821859 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:39.826447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.826422 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6"] Oct 30 22:26:39.861801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.861771 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:39.861977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.861932 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:26:39.903336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.903304 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:39.903501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.903410 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:39.903561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:39.903549 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9nb\" (UniqueName: \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.005520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.005489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.005730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.005671 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9nb\" (UniqueName: \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.005921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.005901 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.006361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.006325 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.008693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.008662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.028086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.027001 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9nb\" (UniqueName: \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.141370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.141337 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:40.677038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:40.672808 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6"] Oct 30 22:26:40.677038 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:26:40.673042 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb918dc_3b43_490a_8049_5c008c73ced4.slice/crio-cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66 WatchSource:0}: Error finding container cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66: Status 404 returned error can't find the container with id cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66 Oct 30 22:26:41.443046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:41.443010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" event={"ID":"0cb918dc-3b43-490a-8049-5c008c73ced4","Type":"ContainerStarted","Data":"fb1541c32dfa1b42c4d254f936b66e9a31634d643fb67c43aeda4f687ebcecba"} Oct 30 22:26:41.443204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:41.443063 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" event={"ID":"0cb918dc-3b43-490a-8049-5c008c73ced4","Type":"ContainerStarted","Data":"cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66"} Oct 30 22:26:41.463569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:41.463522 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" podStartSLOduration=2.46350813 podStartE2EDuration="2.46350813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:41.460852749 +0000 UTC m=+2825.410807696" watchObservedRunningTime="2026-10-30 22:26:41.46350813 +0000 UTC m=+2825.413463076" Oct 30 22:26:42.457670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:42.457632 5036 generic.go:356] "Generic (PLEG): container finished" podID="0cb918dc-3b43-490a-8049-5c008c73ced4" containerID="fb1541c32dfa1b42c4d254f936b66e9a31634d643fb67c43aeda4f687ebcecba" exitCode=0 Oct 30 22:26:42.458140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:42.457767 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" event={"ID":"0cb918dc-3b43-490a-8049-5c008c73ced4","Type":"ContainerDied","Data":"fb1541c32dfa1b42c4d254f936b66e9a31634d643fb67c43aeda4f687ebcecba"} Oct 30 22:26:44.087616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.087539 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:44.227290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.226312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\" (UniqueName: \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\") pod \"0cb918dc-3b43-490a-8049-5c008c73ced4\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " Oct 30 22:26:44.227290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.226422 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\") pod \"0cb918dc-3b43-490a-8049-5c008c73ced4\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " Oct 30 22:26:44.252291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.250187 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb" pod "0cb918dc-3b43-490a-8049-5c008c73ced4" (UID: "0cb918dc-3b43-490a-8049-5c008c73ced4"). InnerVolumeSpecName "kube-api-access-tg9nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:44.266665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.259474 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca" pod "0cb918dc-3b43-490a-8049-5c008c73ced4" (UID: "0cb918dc-3b43-490a-8049-5c008c73ced4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:44.334990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.333479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\") pod \"0cb918dc-3b43-490a-8049-5c008c73ced4\" (UID: \"0cb918dc-3b43-490a-8049-5c008c73ced4\") " Oct 30 22:26:44.334990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.334608 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9nb\" (UniqueName: \"kubernetes.io/projected/0cb918dc-3b43-490a-8049-5c008c73ced4-kube-api-access-tg9nb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:44.334990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.334633 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:44.493586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.493519 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca" pod "0cb918dc-3b43-490a-8049-5c008c73ced4" (UID: "0cb918dc-3b43-490a-8049-5c008c73ced4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:44.553535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.550177 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb918dc-3b43-490a-8049-5c008c73ced4-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:44.583915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.583883 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" Oct 30 22:26:44.583915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.583908 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6" event={"ID":"0cb918dc-3b43-490a-8049-5c008c73ced4","Type":"ContainerDied","Data":"cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66"} Oct 30 22:26:44.584131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.583960 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd82bb22fb58991eff20adf20c13db8b73168e7e9057ddfcd21a9572ab2af66" Oct 30 22:26:44.587155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.587119 5036 generic.go:356] "Generic (PLEG): container finished" podID="66837898-383c-4ac7-93bb-29b5de1a190b" containerID="33435b0726978b771285876da812adc3323c198c02cdf42a20ace5adc2fe53fa" exitCode=0 Oct 30 22:26:44.587341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.587214 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerDied","Data":"33435b0726978b771285876da812adc3323c198c02cdf42a20ace5adc2fe53fa"} Oct 30 22:26:44.587341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:44.587304 5036 scope.go:122] "RemoveContainer" containerID="0216a64334c9859397c9b98badef74818f1d074822cd12d113fb582cde86f208" Oct 30 22:26:45.408290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.405303 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573503 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573634 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573690 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573730 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.573966 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.574056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.574185 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.574247 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\") pod \"66837898-383c-4ac7-93bb-29b5de1a190b\" (UID: \"66837898-383c-4ac7-93bb-29b5de1a190b\") " Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.575362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.575486 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.583043 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "kube-api-access-xpw57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.584110 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.585494 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.585531 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:45.586109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.585969 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:45.586952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.586722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:45.587078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.587052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token" pod "66837898-383c-4ac7-93bb-29b5de1a190b" (UID: "66837898-383c-4ac7-93bb-29b5de1a190b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:45.602737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.602656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" event={"ID":"66837898-383c-4ac7-93bb-29b5de1a190b","Type":"ContainerDied","Data":"e56b87479507e6a280066f0d5a16b1e660b2be4fc1f60969979708482e0ddbdd"} Oct 30 22:26:45.602737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.602705 5036 scope.go:122] "RemoveContainer" containerID="33435b0726978b771285876da812adc3323c198c02cdf42a20ace5adc2fe53fa" Oct 30 22:26:45.602830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.602796 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7" Oct 30 22:26:45.683300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683255 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683305 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683316 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683334 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66837898-383c-4ac7-93bb-29b5de1a190b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683347 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpw57\" (UniqueName: \"kubernetes.io/projected/66837898-383c-4ac7-93bb-29b5de1a190b-kube-api-access-xpw57\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683356 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683365 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683374 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66837898-383c-4ac7-93bb-29b5de1a190b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.683403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.683382 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66837898-383c-4ac7-93bb-29b5de1a190b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:45.705522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.705484 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:26:45.707707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:45.707681 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-rvpf7"] Oct 30 22:26:46.188509 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:46.188469 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:46.200947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:46.200917 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" path="/var/lib/kubelet/pods/66837898-383c-4ac7-93bb-29b5de1a190b/volumes" Oct 30 22:26:48.162074 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:26:48.162006 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:26:49.987646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:49.987596 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:26:49.997863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:49.997833 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:26:52.961439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:52.961400 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:52.961834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:52.961468 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:54.926618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.926584 5036 generic.go:356] "Generic (PLEG): container finished" podID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerID="2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7" exitCode=0 Oct 30 22:26:54.927096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.926655 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerDied","Data":"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7"} Oct 30 22:26:54.927213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.927192 5036 scope.go:122] "RemoveContainer" containerID="2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7" Oct 30 22:26:54.929081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929056 5036 generic.go:356] "Generic (PLEG): container finished" podID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerID="d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6" exitCode=0 Oct 30 22:26:54.929209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929091 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerDied","Data":"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6"} Oct 30 22:26:54.929493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929467 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.929553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929504 5036 scope.go:122] "RemoveContainer" containerID="d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6" Oct 30 22:26:54.929553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929539 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.929626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.929576 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.931386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.931358 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.931499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.931421 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.931499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.931473 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:54.992289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.990361 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" containerID="cri-o://cca817d13b3349540d9fe5d2c236d762aa8aef1fa9535e9db56a7eaafc21e5a9" gracePeriod=25 Oct 30 22:26:54.998685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:54.998633 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" containerID="cri-o://39f6dc0b97f7eb99687502baf0cefe466f74395603e679966732d2f54a2fdcb0" gracePeriod=25 Oct 30 22:26:55.778380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.778341 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6"] Oct 30 22:26:55.916933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.907802 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c8ecf-cndn6"] Oct 30 22:26:55.965376 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.965337 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerStarted","Data":"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3"} Oct 30 22:26:55.972655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.972623 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerStarted","Data":"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1"} Oct 30 22:26:55.975767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.975725 5036 generic.go:356] "Generic (PLEG): container finished" podID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerID="eb99206f09bd7f327938fb280d898de79d7d3121fd64e2b9480b327d2df55090" exitCode=0 Oct 30 22:26:55.975917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:55.975863 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" event={"ID":"6c3fc258-f621-46c3-8d79-efe924508ee5","Type":"ContainerDied","Data":"eb99206f09bd7f327938fb280d898de79d7d3121fd64e2b9480b327d2df55090"} Oct 30 22:26:56.169712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.169683 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:26:56.198116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.198084 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb918dc-3b43-490a-8049-5c008c73ced4" path="/var/lib/kubelet/pods/0cb918dc-3b43-490a-8049-5c008c73ced4/volumes" Oct 30 22:26:56.252779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.252746 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\") pod \"6c3fc258-f621-46c3-8d79-efe924508ee5\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " Oct 30 22:26:56.252975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.252954 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\" (UniqueName: \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\") pod \"6c3fc258-f621-46c3-8d79-efe924508ee5\" (UID: \"6c3fc258-f621-46c3-8d79-efe924508ee5\") " Oct 30 22:26:56.253180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.253152 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume" pod "6c3fc258-f621-46c3-8d79-efe924508ee5" (UID: "6c3fc258-f621-46c3-8d79-efe924508ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:56.253770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.253742 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3fc258-f621-46c3-8d79-efe924508ee5-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.258819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.258749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78" pod "6c3fc258-f621-46c3-8d79-efe924508ee5" (UID: "6c3fc258-f621-46c3-8d79-efe924508ee5"). InnerVolumeSpecName "kube-api-access-lkh78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:56.356953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.356880 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkh78\" (UniqueName: \"kubernetes.io/projected/6c3fc258-f621-46c3-8d79-efe924508ee5-kube-api-access-lkh78\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.396790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.396757 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:26:56.398003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.397983 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:26:56.398063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.398024 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:26:56.497598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.497562 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:26:56.497782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.497657 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:26:56.497782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.497698 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:26:56.992352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.992321 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" Oct 30 22:26:56.993520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.993497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw" event={"ID":"6c3fc258-f621-46c3-8d79-efe924508ee5","Type":"ContainerDied","Data":"7af506e86ef79d6a6a596f95c020ae25d316a4af8cc1a208389e8c2b2d488390"} Oct 30 22:26:56.994364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:56.994344 5036 scope.go:122] "RemoveContainer" containerID="eb99206f09bd7f327938fb280d898de79d7d3121fd64e2b9480b327d2df55090" Oct 30 22:26:57.088531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.088488 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:26:57.093984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.093959 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5313f0-6c5dc48789-pj9bw"] Oct 30 22:26:57.396640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.396602 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:26:57.396835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.396687 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:26:57.507075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.507039 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:26:57.507561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.507518 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:26:57.689589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.689493 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:57.944486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.944383 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:26:57.984686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.984649 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:26:57.986684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.986646 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" Oct 30 22:26:57.986974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.986849 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" Oct 30 22:26:57.987089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987078 5036 cpu_manager.go:414] "containerMap: removing container" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.987196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987184 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.987324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987314 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0cb918dc-3b43-490a-8049-5c008c73ced4" containerName="revoke-keys" Oct 30 22:26:57.987426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987416 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb918dc-3b43-490a-8049-5c008c73ced4" containerName="revoke-keys" Oct 30 22:26:57.987552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987543 5036 cpu_manager.go:414] "containerMap: removing container" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.987689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.987678 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.988903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.988872 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" containerName="payload-processing" Oct 30 22:26:57.989056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.989044 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.989160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.989151 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb918dc-3b43-490a-8049-5c008c73ced4" containerName="revoke-keys" Oct 30 22:26:57.989297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.989286 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="66837898-383c-4ac7-93bb-29b5de1a190b" containerName="istio-proxy" Oct 30 22:26:57.999989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.997372 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:26:57.999989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:57.997485 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.135525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.135615 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.135658 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.135743 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.135947 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.136037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.136142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhhr\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.136302 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.160289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.136330 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.198430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.198355 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3fc258-f621-46c3-8d79-efe924508ee5" path="/var/lib/kubelet/pods/6c3fc258-f621-46c3-8d79-efe924508ee5/volumes" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.238764 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.238903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239019 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhhr\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239145 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239388 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.239484 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.254615 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.257402 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.259566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.259912 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.261288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.260206 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.270488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.270427 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhhr\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.281798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.281766 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.281984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.281915 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.282250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.282229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.400329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.399582 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:26:58.416492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.400904 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:26:58.416492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.400956 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:26:58.505544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.500483 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:26:58.505544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.500556 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:26:58.790637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:58.790424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:59.177932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.177896 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:26:59.185304 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:26:59.185253 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e38752a_b9f0_453f_b884_f15c3522c7b2.slice/crio-66acf984f982b0e230e5b5bbfd1f833adef82756587ed927828bce9275f83e99 WatchSource:0}: Error finding container 66acf984f982b0e230e5b5bbfd1f833adef82756587ed927828bce9275f83e99: Status 404 returned error can't find the container with id 66acf984f982b0e230e5b5bbfd1f833adef82756587ed927828bce9275f83e99 Oct 30 22:26:59.189297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.189240 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:59.189471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.189453 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:59.189568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.189556 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:59.409538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.405436 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:26:59.409538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.405595 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:26:59.497304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.497062 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:26:59.497304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:26:59.497107 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:27:00.177292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.176161 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" event={"ID":"7e38752a-b9f0-453f-b884-f15c3522c7b2","Type":"ContainerStarted","Data":"5a64d815843df9d12cf808d95b3cff3be503ce058bb8ac284af3c81a99d46475"} Oct 30 22:27:00.177292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.176305 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" event={"ID":"7e38752a-b9f0-453f-b884-f15c3522c7b2","Type":"ContainerStarted","Data":"66acf984f982b0e230e5b5bbfd1f833adef82756587ed927828bce9275f83e99"} Oct 30 22:27:00.301294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.295648 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podStartSLOduration=3.295626389 podStartE2EDuration="3.295626389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:00.207403995 +0000 UTC m=+2844.157358966" watchObservedRunningTime="2026-10-30 22:27:00.295626389 +0000 UTC m=+2844.245581336" Oct 30 22:27:00.397017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.396982 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:27:00.397227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.397205 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:27:00.402300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.402024 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:27:00.402635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.402610 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:00.402772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.402653 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:00.497955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.497872 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:27:00.497955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:00.497926 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:27:01.399981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.399948 5036 patch_prober.go:29] interesting pod/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Oct 30 22:27:01.400865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.400559 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Oct 30 22:27:01.400865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.400761 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:01.400865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.400811 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:01.505467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.505425 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 22:27:01.505760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:01.505721 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 22:27:02.401558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.401515 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:02.402000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.401585 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:02.850358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.849606 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:27:02.870294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.860505 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:27:02.931292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.930759 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:27:02.987316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.932235 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:27:02.987316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:02.962026 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:27:03.401693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:03.401659 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:03.402134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:03.401721 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:04.071226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.071183 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:27:04.072492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.072438 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" containerID="cri-o://293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2" gracePeriod=30 Oct 30 22:27:04.412295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.402666 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:04.412295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.402737 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:04.883492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.883459 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl"] Oct 30 22:27:04.888617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.888596 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:04.897012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.896987 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:04.897182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.897142 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:04.897182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.897082 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:27:04.897440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.897200 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:04.898757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.898738 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl"] Oct 30 22:27:04.904063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.904037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:04.904343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.904325 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:04.904411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:04.904352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6p2\" (UniqueName: \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.006233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.006198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.006430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.006399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.006430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.006420 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6p2\" (UniqueName: \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.007299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.007240 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.008872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.008853 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.050299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.050236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6p2\" (UniqueName: \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.207151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.207076 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:05.402436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.402291 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:05.402436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.402350 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:05.622044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:05.621806 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl"] Oct 30 22:27:06.383123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:06.383086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" event={"ID":"66dd0b75-8823-4215-95ff-cd0a0116548c","Type":"ContainerStarted","Data":"bb0454bd9cff5b310563d41613858e511f159871d6c99856d3fac3adbd564eb8"} Oct 30 22:27:06.383455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:06.383141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" event={"ID":"66dd0b75-8823-4215-95ff-cd0a0116548c","Type":"ContainerStarted","Data":"06f49f0768222f7112555e82ca25fabb737a998a2dc90c9713b85f2fcedb8534"} Oct 30 22:27:06.400699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:06.400619 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 22:27:06.400699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:06.400672 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 22:27:06.407670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:06.407621 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" podStartSLOduration=2.407608946 podStartE2EDuration="2.407608946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:06.404079185 +0000 UTC m=+2850.354034131" watchObservedRunningTime="2026-10-30 22:27:06.407608946 +0000 UTC m=+2850.357563891" Oct 30 22:27:07.394120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.394071 5036 generic.go:356] "Generic (PLEG): container finished" podID="66dd0b75-8823-4215-95ff-cd0a0116548c" containerID="bb0454bd9cff5b310563d41613858e511f159871d6c99856d3fac3adbd564eb8" exitCode=0 Oct 30 22:27:07.394539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.394142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" event={"ID":"66dd0b75-8823-4215-95ff-cd0a0116548c","Type":"ContainerDied","Data":"bb0454bd9cff5b310563d41613858e511f159871d6c99856d3fac3adbd564eb8"} Oct 30 22:27:07.406001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.405976 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:27:07.406290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.406255 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:27:07.407082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.407065 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:27:07.686166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:07.686081 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:08.605562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.605523 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:08.705178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.705094 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:08.737157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.737124 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\") pod \"66dd0b75-8823-4215-95ff-cd0a0116548c\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " Oct 30 22:27:08.737487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.737465 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\") pod \"66dd0b75-8823-4215-95ff-cd0a0116548c\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " Oct 30 22:27:08.737573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.737491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca" pod "66dd0b75-8823-4215-95ff-cd0a0116548c" (UID: "66dd0b75-8823-4215-95ff-cd0a0116548c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:08.737639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.737600 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\" (UniqueName: \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\") pod \"66dd0b75-8823-4215-95ff-cd0a0116548c\" (UID: \"66dd0b75-8823-4215-95ff-cd0a0116548c\") " Oct 30 22:27:08.738594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.738567 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:08.740228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.740201 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca" pod "66dd0b75-8823-4215-95ff-cd0a0116548c" (UID: "66dd0b75-8823-4215-95ff-cd0a0116548c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:08.741450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.741418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2" pod "66dd0b75-8823-4215-95ff-cd0a0116548c" (UID: "66dd0b75-8823-4215-95ff-cd0a0116548c"). InnerVolumeSpecName "kube-api-access-rv6p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:08.842565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.841439 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66dd0b75-8823-4215-95ff-cd0a0116548c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:08.842565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:08.841476 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6p2\" (UniqueName: \"kubernetes.io/projected/66dd0b75-8823-4215-95ff-cd0a0116548c-kube-api-access-rv6p2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.366573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.366546 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:27:09.417444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.417409 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" Oct 30 22:27:09.426395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.426357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl" event={"ID":"66dd0b75-8823-4215-95ff-cd0a0116548c","Type":"ContainerDied","Data":"06f49f0768222f7112555e82ca25fabb737a998a2dc90c9713b85f2fcedb8534"} Oct 30 22:27:09.426636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.426423 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f49f0768222f7112555e82ca25fabb737a998a2dc90c9713b85f2fcedb8534" Oct 30 22:27:09.430107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.429657 5036 generic.go:356] "Generic (PLEG): container finished" podID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerID="293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2" exitCode=0 Oct 30 22:27:09.433341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.432472 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" Oct 30 22:27:09.433341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.433154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerDied","Data":"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2"} Oct 30 22:27:09.433341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.433206 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc" event={"ID":"cfe6040c-17a5-4962-aa05-e41be16529c5","Type":"ContainerDied","Data":"64d3de649f1f21e2d3137dc6f89376a0011716f787c95932511b199fdbf3da22"} Oct 30 22:27:09.433341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.433237 5036 scope.go:122] "RemoveContainer" containerID="293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2" Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.479986 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480232 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480460 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480566 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480602 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480671 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480685 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480719 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480841 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.481329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.480904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\") pod \"cfe6040c-17a5-4962-aa05-e41be16529c5\" (UID: \"cfe6040c-17a5-4962-aa05-e41be16529c5\") " Oct 30 22:27:09.501454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.488339 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:09.501454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.491743 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "kube-api-access-8r6m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:09.501454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.492438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:09.501454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.493565 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:09.503694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.502431 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:09.504175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.503860 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:09.509114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.509052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:09.509448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.509373 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo" pod "cfe6040c-17a5-4962-aa05-e41be16529c5" (UID: "cfe6040c-17a5-4962-aa05-e41be16529c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:09.543437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543401 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543431 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543449 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6m4\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-kube-api-access-8r6m4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543469 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe6040c-17a5-4962-aa05-e41be16529c5-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543483 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543498 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543501 5036 scope.go:122] "RemoveContainer" containerID="ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7" Oct 30 22:27:09.543650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.543511 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.547489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.547465 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 22:27:09.557675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.557656 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.563588 5036 scope.go:122] "RemoveContainer" containerID="293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2" Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:09.564860 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2\": container with ID starting with 293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2 not found: ID does not exist" containerID="293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2" Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.564885 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2"} err="failed to get container status \"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2\": rpc error: code = NotFound desc = could not find container \"293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2\": container with ID starting with 293cbef6a9413151b8ac426e6d691469d6642130d9ddee27b715af6626e6dbc2 not found: ID does not exist" Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.564901 5036 scope.go:122] "RemoveContainer" containerID="ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7" Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:09.565365 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7\": container with ID starting with ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7 not found: ID does not exist" containerID="ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7" Oct 30 22:27:09.565423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.565399 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7"} err="failed to get container status \"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7\": rpc error: code = NotFound desc = could not find container \"ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7\": container with ID starting with ed72229d7f03da16dc471371979e3fc6e70ffcda164af43a9b9aaf5171dea0d7 not found: ID does not exist" Oct 30 22:27:09.573166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.572944 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 22:27:09.644899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.644868 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.644899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.644896 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe6040c-17a5-4962-aa05-e41be16529c5-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:09.761769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.761737 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:27:09.777425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:09.777398 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-hfjfc"] Oct 30 22:27:10.192695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:10.192659 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" path="/var/lib/kubelet/pods/cfe6040c-17a5-4962-aa05-e41be16529c5/volumes" Oct 30 22:27:15.049206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.049172 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 22:27:15.085004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.081384 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 22:27:15.716943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.716906 5036 generic.go:356] "Generic (PLEG): container finished" podID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerID="39f6dc0b97f7eb99687502baf0cefe466f74395603e679966732d2f54a2fdcb0" exitCode=0 Oct 30 22:27:15.717342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.717316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" event={"ID":"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d","Type":"ContainerDied","Data":"39f6dc0b97f7eb99687502baf0cefe466f74395603e679966732d2f54a2fdcb0"} Oct 30 22:27:15.728327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.719231 5036 generic.go:356] "Generic (PLEG): container finished" podID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerID="cca817d13b3349540d9fe5d2c236d762aa8aef1fa9535e9db56a7eaafc21e5a9" exitCode=0 Oct 30 22:27:15.728327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.719395 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" event={"ID":"41e10f6d-a52a-41d0-9d15-41c01a8f3a93","Type":"ContainerDied","Data":"cca817d13b3349540d9fe5d2c236d762aa8aef1fa9535e9db56a7eaafc21e5a9"} Oct 30 22:27:15.749869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.749841 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:27:15.765316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.765288 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:27:15.815808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.815773 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\" (UniqueName: \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\") pod \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " Oct 30 22:27:15.816011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.815832 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\") pod \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\" (UID: \"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d\") " Oct 30 22:27:15.816289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.816250 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume" pod "acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" (UID: "acedbaf4-45ce-4ca9-89f9-371f6bcfc44d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.816505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.816480 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\") pod \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " Oct 30 22:27:15.816615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.816527 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\" (UniqueName: \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\") pod \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\" (UID: \"41e10f6d-a52a-41d0-9d15-41c01a8f3a93\") " Oct 30 22:27:15.817123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.817098 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume" pod "41e10f6d-a52a-41d0-9d15-41c01a8f3a93" (UID: "41e10f6d-a52a-41d0-9d15-41c01a8f3a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.818020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.817995 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.818116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.818026 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.819221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.819199 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh" pod "acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" (UID: "acedbaf4-45ce-4ca9-89f9-371f6bcfc44d"). InnerVolumeSpecName "kube-api-access-hr9wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:15.821168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.821142 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2" pod "41e10f6d-a52a-41d0-9d15-41c01a8f3a93" (UID: "41e10f6d-a52a-41d0-9d15-41c01a8f3a93"). InnerVolumeSpecName "kube-api-access-tkdw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:15.929292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.922056 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9wh\" (UniqueName: \"kubernetes.io/projected/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d-kube-api-access-hr9wh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.929292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:15.922101 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdw2\" (UniqueName: \"kubernetes.io/projected/41e10f6d-a52a-41d0-9d15-41c01a8f3a93-kube-api-access-tkdw2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.731621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.731592 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" Oct 30 22:27:16.732173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.731591 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq" event={"ID":"41e10f6d-a52a-41d0-9d15-41c01a8f3a93","Type":"ContainerDied","Data":"97c9e461381eb1fd5b93804d3638e3fa9ee7c9e3acfa317c488c37a56bf61bc4"} Oct 30 22:27:16.732406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.732310 5036 scope.go:122] "RemoveContainer" containerID="cca817d13b3349540d9fe5d2c236d762aa8aef1fa9535e9db56a7eaafc21e5a9" Oct 30 22:27:16.733757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.733584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" event={"ID":"acedbaf4-45ce-4ca9-89f9-371f6bcfc44d","Type":"ContainerDied","Data":"98fa59e13dee22000fced512270d3f36864ac4c4a6efbe71e5843622bdba08d4"} Oct 30 22:27:16.733757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.733663 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m" Oct 30 22:27:16.748370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.748346 5036 scope.go:122] "RemoveContainer" containerID="39f6dc0b97f7eb99687502baf0cefe466f74395603e679966732d2f54a2fdcb0" Oct 30 22:27:16.760783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.760752 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:27:16.767081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.767058 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c8ecf-598fb9ccc9-cbf8m"] Oct 30 22:27:16.774649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.774629 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:27:16.781771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:16.781743 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c8ecf-7b56779f4c-4fkqq"] Oct 30 22:27:18.100156 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:18.100117 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:27:18.197038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:18.197000 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" path="/var/lib/kubelet/pods/41e10f6d-a52a-41d0-9d15-41c01a8f3a93/volumes" Oct 30 22:27:18.197546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:18.197528 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" path="/var/lib/kubelet/pods/acedbaf4-45ce-4ca9-89f9-371f6bcfc44d/volumes" Oct 30 22:27:20.049906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:20.049841 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" containerID="cri-o://5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa" gracePeriod=25 Oct 30 22:27:20.082547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:20.082511 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" containerID="cri-o://fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545" gracePeriod=25 Oct 30 22:27:24.580959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.580913 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" containerID="cri-o://25bfff488cc8ae97e6f99adcae1fadc08bd76934c978e2193a8cd8169954eb0b" gracePeriod=45 Oct 30 22:27:24.628600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.628566 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" containerID="cri-o://065b6d83a6fe6230f4a7ec1fab41c230cc60271ca5d90bbfa068ae7cc0f4ca58" gracePeriod=45 Oct 30 22:27:24.629514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.629483 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" containerID="cri-o://eeb7338ecc893527a391a6ea831c335eee9a3007865f6f8b343f444024b843f5" gracePeriod=45 Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.932498 5036 generic.go:356] "Generic (PLEG): container finished" podID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerID="065b6d83a6fe6230f4a7ec1fab41c230cc60271ca5d90bbfa068ae7cc0f4ca58" exitCode=0 Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.932620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" event={"ID":"5e9233a1-3653-4889-a319-1dda932f5c0a","Type":"ContainerDied","Data":"065b6d83a6fe6230f4a7ec1fab41c230cc60271ca5d90bbfa068ae7cc0f4ca58"} Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.934728 5036 generic.go:356] "Generic (PLEG): container finished" podID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerID="eeb7338ecc893527a391a6ea831c335eee9a3007865f6f8b343f444024b843f5" exitCode=0 Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.934909 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" event={"ID":"f80e61a3-945b-4f1f-a486-3c93cffb3a64","Type":"ContainerDied","Data":"eeb7338ecc893527a391a6ea831c335eee9a3007865f6f8b343f444024b843f5"} Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.936732 5036 generic.go:356] "Generic (PLEG): container finished" podID="e9f550c0-8573-46f1-9507-3d16da78855c" containerID="25bfff488cc8ae97e6f99adcae1fadc08bd76934c978e2193a8cd8169954eb0b" exitCode=0 Oct 30 22:27:24.938290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:24.936768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" event={"ID":"e9f550c0-8573-46f1-9507-3d16da78855c","Type":"ContainerDied","Data":"25bfff488cc8ae97e6f99adcae1fadc08bd76934c978e2193a8cd8169954eb0b"} Oct 30 22:27:25.118876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.118850 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:27:25.132563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.132050 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:27:25.137594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.137573 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193357 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193575 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193658 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\" (UniqueName: \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193716 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\" (UniqueName: \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193750 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193784 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\" (UniqueName: \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193893 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193942 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.193979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194047 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194121 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\") pod \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\" (UID: \"f80e61a3-945b-4f1f-a486-3c93cffb3a64\") " Oct 30 22:27:25.194247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194167 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.195145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194281 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.195145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.195145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\") pod \"e9f550c0-8573-46f1-9507-3d16da78855c\" (UID: \"e9f550c0-8573-46f1-9507-3d16da78855c\") " Oct 30 22:27:25.195145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.194491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\") pod \"5e9233a1-3653-4889-a319-1dda932f5c0a\" (UID: \"5e9233a1-3653-4889-a319-1dda932f5c0a\") " Oct 30 22:27:25.195746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.195718 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.197327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.196893 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.197327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.197239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.197521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.197496 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.197908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.197880 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.200141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.200112 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.201007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.200983 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.201433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.201411 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.202390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.202370 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.202868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.202678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.202976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.202949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.203809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.203187 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.207501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.207471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "kube-api-access-9l24w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:25.207869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.207849 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "kube-api-access-fhptm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:25.208857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.208829 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "kube-api-access-vlzgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:25.209663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.209640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs" pod "5e9233a1-3653-4889-a319-1dda932f5c0a" (UID: "5e9233a1-3653-4889-a319-1dda932f5c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:25.209807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.209784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:25.210716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.210691 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm" pod "e9f550c0-8573-46f1-9507-3d16da78855c" (UID: "e9f550c0-8573-46f1-9507-3d16da78855c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:25.211443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.211419 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs" pod "f80e61a3-945b-4f1f-a486-3c93cffb3a64" (UID: "f80e61a3-945b-4f1f-a486-3c93cffb3a64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301124 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301166 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301181 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301195 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301208 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f550c0-8573-46f1-9507-3d16da78855c-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301221 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e9233a1-3653-4889-a319-1dda932f5c0a-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301236 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l24w\" (UniqueName: \"kubernetes.io/projected/f80e61a3-945b-4f1f-a486-3c93cffb3a64-kube-api-access-9l24w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301251 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80e61a3-945b-4f1f-a486-3c93cffb3a64-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301305 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301321 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/5e9233a1-3653-4889-a319-1dda932f5c0a-kube-api-access-fhptm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301343 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzgl\" (UniqueName: \"kubernetes.io/projected/e9f550c0-8573-46f1-9507-3d16da78855c-kube-api-access-vlzgl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301359 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9233a1-3653-4889-a319-1dda932f5c0a-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301374 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301390 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301403 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301417 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80e61a3-945b-4f1f-a486-3c93cffb3a64-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.301805 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.301429 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f550c0-8573-46f1-9507-3d16da78855c-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:25.510054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.509922 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:25.510054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.509996 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:25.875215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.875125 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:25.948219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.948181 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" Oct 30 22:27:25.948449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.948217 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt" event={"ID":"e9f550c0-8573-46f1-9507-3d16da78855c","Type":"ContainerDied","Data":"c902655f3e9dd371166e940fce35107a89333883be48bc085925a032d1382944"} Oct 30 22:27:25.948449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.948312 5036 scope.go:122] "RemoveContainer" containerID="25bfff488cc8ae97e6f99adcae1fadc08bd76934c978e2193a8cd8169954eb0b" Oct 30 22:27:25.950357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.950337 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" Oct 30 22:27:25.950500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.950354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj" event={"ID":"5e9233a1-3653-4889-a319-1dda932f5c0a","Type":"ContainerDied","Data":"21a1d10bcd1e390c15d27516120939902537f03328734e60587eb766cae43a11"} Oct 30 22:27:25.952241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.952207 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" event={"ID":"f80e61a3-945b-4f1f-a486-3c93cffb3a64","Type":"ContainerDied","Data":"8170018d05757ed1515d1c90ad4a3b06dad4f2a1ca8c4d60ab1e11e6952c14b9"} Oct 30 22:27:25.952405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.952296 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj" Oct 30 22:27:25.965141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.965123 5036 scope.go:122] "RemoveContainer" containerID="065b6d83a6fe6230f4a7ec1fab41c230cc60271ca5d90bbfa068ae7cc0f4ca58" Oct 30 22:27:25.976416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:25.976401 5036 scope.go:122] "RemoveContainer" containerID="eeb7338ecc893527a391a6ea831c335eee9a3007865f6f8b343f444024b843f5" Oct 30 22:27:26.004234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.004201 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 22:27:26.009077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.009056 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-w2-2b2e72-kserve-669fc48dcf-jknzt"] Oct 30 22:27:26.042132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.042087 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 22:27:26.049427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.049396 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/simulator-w2-2b2e72-kserve-76ffd77749-wdgfj"] Oct 30 22:27:26.063065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.063041 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 22:27:26.069861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.069838 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/premium-w2-2b2e72-kserve-7dd7f57f5b-zszjj"] Oct 30 22:27:26.198025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.197981 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" path="/var/lib/kubelet/pods/5e9233a1-3653-4889-a319-1dda932f5c0a/volumes" Oct 30 22:27:26.198837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.198809 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" path="/var/lib/kubelet/pods/e9f550c0-8573-46f1-9507-3d16da78855c/volumes" Oct 30 22:27:26.201069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.201044 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" path="/var/lib/kubelet/pods/f80e61a3-945b-4f1f-a486-3c93cffb3a64/volumes" Oct 30 22:27:26.201711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.201687 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4"] Oct 30 22:27:26.202887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202862 5036 cpu_manager.go:414] "containerMap: removing container" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" Oct 30 22:27:26.202887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202886 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202923 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202932 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202943 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202949 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202975 5036 cpu_manager.go:414] "containerMap: removing container" podUID="66dd0b75-8823-4215-95ff-cd0a0116548c" containerName="revoke-keys" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.202982 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66dd0b75-8823-4215-95ff-cd0a0116548c" containerName="revoke-keys" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203002 5036 cpu_manager.go:414] "containerMap: removing container" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203009 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203026 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203033 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203049 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203056 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203076 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203083 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203562 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80e61a3-945b-4f1f-a486-3c93cffb3a64" containerName="main" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203582 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203597 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="66dd0b75-8823-4215-95ff-cd0a0116548c" containerName="revoke-keys" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203610 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="acedbaf4-45ce-4ca9-89f9-371f6bcfc44d" containerName="payload-pre-processing" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203623 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f550c0-8573-46f1-9507-3d16da78855c" containerName="main" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203636 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9233a1-3653-4889-a319-1dda932f5c0a" containerName="main" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203648 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe6040c-17a5-4962-aa05-e41be16529c5" containerName="istio-proxy" Oct 30 22:27:26.203957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.203666 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e10f6d-a52a-41d0-9d15-41c01a8f3a93" containerName="payload-processing" Oct 30 22:27:26.209747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.209725 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg"] Oct 30 22:27:26.209874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.209855 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.215987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.215963 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:26.216094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216062 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4"] Oct 30 22:27:26.216156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216101 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg"] Oct 30 22:27:26.216210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216184 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.216500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216476 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:26.216588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216537 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:26.216650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.216594 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:27:26.331048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.331234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331124 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.331234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331175 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclpc\" (UniqueName: \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.331511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331487 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27ph\" (UniqueName: \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.331648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331625 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.331778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.331758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.433247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433207 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.433428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.433516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433432 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.433516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclpc\" (UniqueName: \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.433725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433707 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27ph\" (UniqueName: \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.433830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.433797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.434224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.434202 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.434352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.434234 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.436130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.436102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.436713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.436691 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.449432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.449370 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27ph\" (UniqueName: \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\") pod \"maas-api-revoke-sub-b9f7c4e5-prrs4\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.450200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.450180 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclpc\" (UniqueName: \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\") pod \"maas-api-revoke-sub-1f7cd933-trrhg\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.544305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.543511 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:26.556290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.554546 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:26.889443 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:27:26.889235 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c40c3f_32db_41a8_bff0_01000e6f237a.slice/crio-804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7 WatchSource:0}: Error finding container 804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7: Status 404 returned error can't find the container with id 804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7 Oct 30 22:27:26.890109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.890082 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4"] Oct 30 22:27:26.892925 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:27:26.892898 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6830ec2a_d76d_4490_ada9_b9633d7b1ed8.slice/crio-b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83 WatchSource:0}: Error finding container b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83: Status 404 returned error can't find the container with id b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83 Oct 30 22:27:26.896320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.896257 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg"] Oct 30 22:27:26.971072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.971039 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" event={"ID":"35c40c3f-32db-41a8-bff0-01000e6f237a","Type":"ContainerStarted","Data":"804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7"} Oct 30 22:27:26.972634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:26.972562 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" event={"ID":"6830ec2a-d76d-4490-ada9-b9633d7b1ed8","Type":"ContainerStarted","Data":"b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83"} Oct 30 22:27:28.105314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:28.097896 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" event={"ID":"35c40c3f-32db-41a8-bff0-01000e6f237a","Type":"ContainerStarted","Data":"9766521ddd7547fc975d9928b1b7ebc73dcc45d8df4f0f67d19b321517b2eb10"} Oct 30 22:27:28.118789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:28.118750 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" event={"ID":"6830ec2a-d76d-4490-ada9-b9633d7b1ed8","Type":"ContainerStarted","Data":"e423c9aad3c149397dc41c0fcd9b5a23de66fbf929b1b609e130281659748077"} Oct 30 22:27:28.387292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:28.373710 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" podStartSLOduration=2.373689265 podStartE2EDuration="2.373689265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:28.203769609 +0000 UTC m=+2872.153724557" watchObservedRunningTime="2026-10-30 22:27:28.373689265 +0000 UTC m=+2872.323644213" Oct 30 22:27:28.433290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:28.424967 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" podStartSLOduration=2.424942149 podStartE2EDuration="2.424942149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:28.341857804 +0000 UTC m=+2872.291812751" watchObservedRunningTime="2026-10-30 22:27:28.424942149 +0000 UTC m=+2872.374897096" Oct 30 22:27:29.245110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.244337 5036 generic.go:356] "Generic (PLEG): container finished" podID="35c40c3f-32db-41a8-bff0-01000e6f237a" containerID="9766521ddd7547fc975d9928b1b7ebc73dcc45d8df4f0f67d19b321517b2eb10" exitCode=0 Oct 30 22:27:29.245110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.244468 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" event={"ID":"35c40c3f-32db-41a8-bff0-01000e6f237a","Type":"ContainerDied","Data":"9766521ddd7547fc975d9928b1b7ebc73dcc45d8df4f0f67d19b321517b2eb10"} Oct 30 22:27:29.299425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.298524 5036 generic.go:356] "Generic (PLEG): container finished" podID="6830ec2a-d76d-4490-ada9-b9633d7b1ed8" containerID="e423c9aad3c149397dc41c0fcd9b5a23de66fbf929b1b609e130281659748077" exitCode=0 Oct 30 22:27:29.299425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.298680 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" event={"ID":"6830ec2a-d76d-4490-ada9-b9633d7b1ed8","Type":"ContainerDied","Data":"e423c9aad3c149397dc41c0fcd9b5a23de66fbf929b1b609e130281659748077"} Oct 30 22:27:29.688295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.688235 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:29.774637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.774364 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:29.808289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:29.800998 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:30.709429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.709402 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:30.740662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.740566 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:30.782931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.782894 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\") pod \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " Oct 30 22:27:30.783131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.783111 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\" (UniqueName: \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\") pod \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " Oct 30 22:27:30.783252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.783235 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\") pod \"35c40c3f-32db-41a8-bff0-01000e6f237a\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " Oct 30 22:27:30.783373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.783259 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca" pod "6830ec2a-d76d-4490-ada9-b9633d7b1ed8" (UID: "6830ec2a-d76d-4490-ada9-b9633d7b1ed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:30.784127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.784103 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\") pod \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\" (UID: \"6830ec2a-d76d-4490-ada9-b9633d7b1ed8\") " Oct 30 22:27:30.784244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.784157 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\") pod \"35c40c3f-32db-41a8-bff0-01000e6f237a\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " Oct 30 22:27:30.784244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.784193 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\" (UniqueName: \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\") pod \"35c40c3f-32db-41a8-bff0-01000e6f237a\" (UID: \"35c40c3f-32db-41a8-bff0-01000e6f237a\") " Oct 30 22:27:30.785184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.784813 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca" pod "35c40c3f-32db-41a8-bff0-01000e6f237a" (UID: "35c40c3f-32db-41a8-bff0-01000e6f237a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:30.785184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.785118 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.785184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.785142 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.786237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.786206 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca" pod "35c40c3f-32db-41a8-bff0-01000e6f237a" (UID: "35c40c3f-32db-41a8-bff0-01000e6f237a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:30.786933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.786906 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca" pod "6830ec2a-d76d-4490-ada9-b9633d7b1ed8" (UID: "6830ec2a-d76d-4490-ada9-b9633d7b1ed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:30.787185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.787159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph" pod "35c40c3f-32db-41a8-bff0-01000e6f237a" (UID: "35c40c3f-32db-41a8-bff0-01000e6f237a"). InnerVolumeSpecName "kube-api-access-l27ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:30.788845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.788819 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc" pod "6830ec2a-d76d-4490-ada9-b9633d7b1ed8" (UID: "6830ec2a-d76d-4490-ada9-b9633d7b1ed8"). InnerVolumeSpecName "kube-api-access-xclpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:30.886844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.886753 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.886844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.886786 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27ph\" (UniqueName: \"kubernetes.io/projected/35c40c3f-32db-41a8-bff0-01000e6f237a-kube-api-access-l27ph\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.886844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.886801 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xclpc\" (UniqueName: \"kubernetes.io/projected/6830ec2a-d76d-4490-ada9-b9633d7b1ed8-kube-api-access-xclpc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.886844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:30.886813 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c40c3f-32db-41a8-bff0-01000e6f237a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:31.392885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.392856 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" Oct 30 22:27:31.392885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.392863 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4" event={"ID":"35c40c3f-32db-41a8-bff0-01000e6f237a","Type":"ContainerDied","Data":"804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7"} Oct 30 22:27:31.393171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.392926 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804b9d624bc176a4e5c4da9c5a8d8dcaad0fd93f573ad693b6adb82d0b6c99a7" Oct 30 22:27:31.394688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.394657 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" Oct 30 22:27:31.394833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.394700 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg" event={"ID":"6830ec2a-d76d-4490-ada9-b9633d7b1ed8","Type":"ContainerDied","Data":"b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83"} Oct 30 22:27:31.394833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:31.394748 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b358f1c1db8a5d4f5c77d7390e2b8de2958933aa408f89ce9ba6efdd3ac20d83" Oct 30 22:27:34.346470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.346437 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c"] Oct 30 22:27:34.347819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.347791 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35c40c3f-32db-41a8-bff0-01000e6f237a" containerName="revoke-keys" Oct 30 22:27:34.347994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.347978 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c40c3f-32db-41a8-bff0-01000e6f237a" containerName="revoke-keys" Oct 30 22:27:34.348141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.348131 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6830ec2a-d76d-4490-ada9-b9633d7b1ed8" containerName="revoke-keys" Oct 30 22:27:34.348197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.348190 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6830ec2a-d76d-4490-ada9-b9633d7b1ed8" containerName="revoke-keys" Oct 30 22:27:34.348492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.348476 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c40c3f-32db-41a8-bff0-01000e6f237a" containerName="revoke-keys" Oct 30 22:27:34.348597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.348588 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6830ec2a-d76d-4490-ada9-b9633d7b1ed8" containerName="revoke-keys" Oct 30 22:27:34.352817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.352795 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.357584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.357530 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:34.358108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.357649 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:34.358108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.357957 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:34.362109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.361948 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c"] Oct 30 22:27:34.428289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.411212 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmdz\" (UniqueName: \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.428289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.411529 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.428289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.411560 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.456685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.456658 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:27:34.528287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.517573 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.528287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.517626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.528287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.517853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmdz\" (UniqueName: \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.528287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.519073 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.539289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.532692 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.539289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.534898 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmdz\" (UniqueName: \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\") pod \"maas-api-revoke-sub-ec933107-92c6c\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:34.680489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:34.680457 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:35.051722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:35.051510 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c"] Oct 30 22:27:35.469041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:35.469000 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" event={"ID":"4a843eae-17dc-41fa-83ed-f9172d4a1e0e","Type":"ContainerStarted","Data":"62f2c29d4d0cd8a941dfc97b4bbf0000c0ac1d6e0547efcb1ef34120f0ded5c9"} Oct 30 22:27:35.509624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:35.509586 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:35.509840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:35.509818 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:27:36.481551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:36.481471 5036 generic.go:356] "Generic (PLEG): container finished" podID="4a843eae-17dc-41fa-83ed-f9172d4a1e0e" containerID="24ec0181af1bc004821074b4d9cb88cd2063ce94cf350e10f1094483abcaddab" exitCode=0 Oct 30 22:27:36.481903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:36.481553 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" event={"ID":"4a843eae-17dc-41fa-83ed-f9172d4a1e0e","Type":"ContainerDied","Data":"24ec0181af1bc004821074b4d9cb88cd2063ce94cf350e10f1094483abcaddab"} Oct 30 22:27:36.948456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:36.948414 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:36.948807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:36.948490 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:37.709390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.709358 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:37.789841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.789746 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\" (UniqueName: \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\") pod \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " Oct 30 22:27:37.790108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.790080 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\") pod \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " Oct 30 22:27:37.790713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.790252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\") pod \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\" (UID: \"4a843eae-17dc-41fa-83ed-f9172d4a1e0e\") " Oct 30 22:27:37.790713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.790658 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca" pod "4a843eae-17dc-41fa-83ed-f9172d4a1e0e" (UID: "4a843eae-17dc-41fa-83ed-f9172d4a1e0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:37.791203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.791180 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:37.792942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.792913 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz" pod "4a843eae-17dc-41fa-83ed-f9172d4a1e0e" (UID: "4a843eae-17dc-41fa-83ed-f9172d4a1e0e"). InnerVolumeSpecName "kube-api-access-cpmdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:37.793194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.793165 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca" pod "4a843eae-17dc-41fa-83ed-f9172d4a1e0e" (UID: "4a843eae-17dc-41fa-83ed-f9172d4a1e0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:37.937337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.909809 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmdz\" (UniqueName: \"kubernetes.io/projected/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-kube-api-access-cpmdz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:37.937337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:37.909853 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a843eae-17dc-41fa-83ed-f9172d4a1e0e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:38.530435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:38.505665 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" event={"ID":"4a843eae-17dc-41fa-83ed-f9172d4a1e0e","Type":"ContainerDied","Data":"62f2c29d4d0cd8a941dfc97b4bbf0000c0ac1d6e0547efcb1ef34120f0ded5c9"} Oct 30 22:27:38.530435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:38.505738 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f2c29d4d0cd8a941dfc97b4bbf0000c0ac1d6e0547efcb1ef34120f0ded5c9" Oct 30 22:27:38.530435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:38.505817 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c" Oct 30 22:27:40.398216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.398182 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 22:27:40.418042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.417770 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 22:27:40.458289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.457075 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\") pod \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " Oct 30 22:27:40.458289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.457152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\") pod \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " Oct 30 22:27:40.458289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.457373 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\" (UniqueName: \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\") pod \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\" (UID: \"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2\") " Oct 30 22:27:40.458289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.457448 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\" (UniqueName: \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\") pod \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\" (UID: \"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c\") " Oct 30 22:27:40.459289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.459059 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume" pod "cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" (UID: "cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:40.459857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.459833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume" pod "12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" (UID: "12c0f2c3-c3b2-47b1-97c7-ef2762218bf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:40.460752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.460727 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk" pod "cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" (UID: "cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c"). InnerVolumeSpecName "kube-api-access-52srk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:40.461629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.461604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm" pod "12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" (UID: "12c0f2c3-c3b2-47b1-97c7-ef2762218bf2"). InnerVolumeSpecName "kube-api-access-z4rkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:40.592079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.589594 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rkm\" (UniqueName: \"kubernetes.io/projected/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-kube-api-access-z4rkm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:40.592079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.589630 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52srk\" (UniqueName: \"kubernetes.io/projected/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-kube-api-access-52srk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:40.592079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.589644 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:40.592079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.589658 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:40.606692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.606647 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerID="0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e" exitCode=0 Oct 30 22:27:40.606952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.606927 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerDied","Data":"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e"} Oct 30 22:27:40.607592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.607566 5036 scope.go:122] "RemoveContainer" containerID="0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e" Oct 30 22:27:40.610992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.610944 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:40.611174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611144 5036 generic.go:356] "Generic (PLEG): container finished" podID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerID="5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa" exitCode=0 Oct 30 22:27:40.611338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" event={"ID":"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2","Type":"ContainerDied","Data":"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa"} Oct 30 22:27:40.611433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611375 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" event={"ID":"12c0f2c3-c3b2-47b1-97c7-ef2762218bf2","Type":"ContainerDied","Data":"ff35d559d755bb18fd508a96e8e2010bb343ddac67cf57025c642d8234b31f6d"} Oct 30 22:27:40.611433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611422 5036 scope.go:122] "RemoveContainer" containerID="5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa" Oct 30 22:27:40.611610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611594 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g" Oct 30 22:27:40.611690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611589 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:40.611872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.611848 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:40.613842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.613639 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerID="fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545" exitCode=0 Oct 30 22:27:40.614372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.614300 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" event={"ID":"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c","Type":"ContainerDied","Data":"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545"} Oct 30 22:27:40.614372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.614359 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" event={"ID":"cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c","Type":"ContainerDied","Data":"d0db9df36a8cb836f012b0ad84aa2e27d125781f3f3efe0cb588b79ebbfa9f03"} Oct 30 22:27:40.614536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.614317 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds" Oct 30 22:27:40.629080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.629061 5036 scope.go:122] "RemoveContainer" containerID="5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa" Oct 30 22:27:40.629760 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:40.629736 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa\": container with ID starting with 5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa not found: ID does not exist" containerID="5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa" Oct 30 22:27:40.629878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.629769 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa"} err="failed to get container status \"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa\": rpc error: code = NotFound desc = could not find container \"5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa\": container with ID starting with 5e08f7471cdb2179530f5a79d94b39f9d34f877b00004fd5e7f966ea73b95aaa not found: ID does not exist" Oct 30 22:27:40.629878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.629792 5036 scope.go:122] "RemoveContainer" containerID="fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545" Oct 30 22:27:40.645751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.645731 5036 scope.go:122] "RemoveContainer" containerID="fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545" Oct 30 22:27:40.646077 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:40.646048 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545\": container with ID starting with fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545 not found: ID does not exist" containerID="fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545" Oct 30 22:27:40.646172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.646091 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545"} err="failed to get container status \"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545\": rpc error: code = NotFound desc = could not find container \"fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545\": container with ID starting with fff5bdb5c89733aa84b7e8969326700781a5263aa1dd29dccbab5951c2026545 not found: ID does not exist" Oct 30 22:27:40.663004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.662976 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 22:27:40.673916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.673221 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b2e72-7bdbb6ccff-kk98g"] Oct 30 22:27:40.689507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.689119 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 22:27:40.689507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:40.689171 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b2e72-7fb8c47559-qzrds"] Oct 30 22:27:41.653899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:41.653862 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerStarted","Data":"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608"} Oct 30 22:27:42.189835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:42.189801 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" path="/var/lib/kubelet/pods/12c0f2c3-c3b2-47b1-97c7-ef2762218bf2/volumes" Oct 30 22:27:42.190335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:42.190321 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" path="/var/lib/kubelet/pods/cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c/volumes" Oct 30 22:27:42.595129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:42.595053 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:27:42.595296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:42.595239 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:42.595346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:42.595321 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:43.594833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:43.594805 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:43.595241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:43.594868 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.157169 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158472 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4a843eae-17dc-41fa-83ed-f9172d4a1e0e" containerName="revoke-keys" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158493 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a843eae-17dc-41fa-83ed-f9172d4a1e0e" containerName="revoke-keys" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158524 5036 cpu_manager.go:414] "containerMap: removing container" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158532 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158555 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158563 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158807 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a843eae-17dc-41fa-83ed-f9172d4a1e0e" containerName="revoke-keys" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158824 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c0f2c3-c3b2-47b1-97c7-ef2762218bf2" containerName="payload-processing" Oct 30 22:27:44.160290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.158837 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9dd224-b3e8-49bd-af8c-191e1f9d5b6c" containerName="payload-pre-processing" Oct 30 22:27:44.165350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.164622 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.168592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.168565 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:27:44.297999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.297962 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298013 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298234 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298292 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298439 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scln4\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298850 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.298999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.298959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400483 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400675 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400700 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400824 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scln4\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.400923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.400859 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.401108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.401198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401124 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.401198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401168 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.401710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401655 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.402419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401924 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.402419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.402419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.401980 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.402419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.402377 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.403434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.403410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.403776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.403759 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.432691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.428110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.433568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.433519 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scln4\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.484500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.484454 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:44.597814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.597312 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:44.597814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.597374 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:44.944742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.944691 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:27:44.945394 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:27:44.945362 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75903261_86d8_4087_9fa3_6bc5e1d92e8c.slice/crio-b057638d8557fd2205bab653c16e18f627baf10417ac42e2274bd19fceac7a6c WatchSource:0}: Error finding container b057638d8557fd2205bab653c16e18f627baf10417ac42e2274bd19fceac7a6c: Status 404 returned error can't find the container with id b057638d8557fd2205bab653c16e18f627baf10417ac42e2274bd19fceac7a6c Oct 30 22:27:44.949279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.949216 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:44.949398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.949326 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:44.949471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.949408 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:44.954392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.954369 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w"] Oct 30 22:27:44.970054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.966311 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w"] Oct 30 22:27:44.970054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.966435 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:44.974718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.974457 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:27:44.974718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.974663 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:44.974922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.974758 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:44.986643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:44.986614 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:45.138291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.137565 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7c64\" (UniqueName: \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.138291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.137735 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.138291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.137824 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.239291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.239186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.239487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.239467 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7c64\" (UniqueName: \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.239618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.239600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.240829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.240799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.242818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.242796 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.252701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.252680 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7c64\" (UniqueName: \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\") pod \"maas-api-revoke-sub-13be5ba7-l9d6w\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.291458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.291426 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:45.614564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.614473 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:45.614564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.614534 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:45.643348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.643193 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w"] Oct 30 22:27:45.786158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.785934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" event={"ID":"75903261-86d8-4087-9fa3-6bc5e1d92e8c","Type":"ContainerStarted","Data":"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787"} Oct 30 22:27:45.786158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.785987 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" event={"ID":"75903261-86d8-4087-9fa3-6bc5e1d92e8c","Type":"ContainerStarted","Data":"b057638d8557fd2205bab653c16e18f627baf10417ac42e2274bd19fceac7a6c"} Oct 30 22:27:45.787581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.787536 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" event={"ID":"54d26225-aa3f-4a5f-a4f2-3c116cafe885","Type":"ContainerStarted","Data":"8815fc01f213a107d840775e91fcee457c2743cd5d8e186f7d81a0c5d9f2aba2"} Oct 30 22:27:45.869970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:45.865146 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podStartSLOduration=1.865123342 podStartE2EDuration="1.865123342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:45.807739633 +0000 UTC m=+2889.757694594" watchObservedRunningTime="2026-10-30 22:27:45.865123342 +0000 UTC m=+2889.815078289" Oct 30 22:27:46.508288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:46.490495 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:46.508288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:46.493249 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:46.508288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:46.493329 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:46.602349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:46.596010 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:46.602349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:46.596099 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:47.241169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.235404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" event={"ID":"54d26225-aa3f-4a5f-a4f2-3c116cafe885","Type":"ContainerStarted","Data":"f1f70128b1cb7afa55a7971552243cd6906f70881ceea08f8da30dd895cb6378"} Oct 30 22:27:47.417289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.416897 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" podStartSLOduration=3.416874023 podStartE2EDuration="3.416874023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:47.303742418 +0000 UTC m=+2891.253697366" watchObservedRunningTime="2026-10-30 22:27:47.416874023 +0000 UTC m=+2891.366828971" Oct 30 22:27:47.520564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.520499 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:47.520921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.520826 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:47.598372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.598309 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body= Oct 30 22:27:47.598372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:47.598364 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" Oct 30 22:27:48.103623 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:27:48.103591 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:27:48.262219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.261810 5036 generic.go:356] "Generic (PLEG): container finished" podID="54d26225-aa3f-4a5f-a4f2-3c116cafe885" containerID="f1f70128b1cb7afa55a7971552243cd6906f70881ceea08f8da30dd895cb6378" exitCode=0 Oct 30 22:27:48.262219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.261982 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" event={"ID":"54d26225-aa3f-4a5f-a4f2-3c116cafe885","Type":"ContainerDied","Data":"f1f70128b1cb7afa55a7971552243cd6906f70881ceea08f8da30dd895cb6378"} Oct 30 22:27:48.485774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.485733 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:48.486001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.485796 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:48.601092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.601064 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:27:48.601447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.601423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:27:48.602252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:48.602222 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:27:49.485606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.485570 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:49.486064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.485639 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:49.495833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.495806 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:49.604350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.603199 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\") pod \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " Oct 30 22:27:49.606306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.604788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\") pod \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " Oct 30 22:27:49.606306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.604869 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\" (UniqueName: \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\") pod \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\" (UID: \"54d26225-aa3f-4a5f-a4f2-3c116cafe885\") " Oct 30 22:27:49.611133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.611086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca" pod "54d26225-aa3f-4a5f-a4f2-3c116cafe885" (UID: "54d26225-aa3f-4a5f-a4f2-3c116cafe885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:49.623307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.621113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca" pod "54d26225-aa3f-4a5f-a4f2-3c116cafe885" (UID: "54d26225-aa3f-4a5f-a4f2-3c116cafe885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:49.623307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.621932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64" pod "54d26225-aa3f-4a5f-a4f2-3c116cafe885" (UID: "54d26225-aa3f-4a5f-a4f2-3c116cafe885"). InnerVolumeSpecName "kube-api-access-c7c64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:49.685618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.685572 5036 scope.go:122] "RemoveContainer" containerID="62f26994718f5b7d3bfecb3550e9bad1d894828203ec48971f35a71568f612ca" Oct 30 22:27:49.704701 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.704673 5036 scope.go:122] "RemoveContainer" containerID="e9b257a95e7ddd3e671c9c8924e3d0a6eba7d9bf8c13fb8fede3ebb91579ec93" Oct 30 22:27:49.708454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.708417 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7c64\" (UniqueName: \"kubernetes.io/projected/54d26225-aa3f-4a5f-a4f2-3c116cafe885-kube-api-access-c7c64\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:49.708556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.708459 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:49.708556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:49.708476 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d26225-aa3f-4a5f-a4f2-3c116cafe885-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:50.461345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.451583 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" Oct 30 22:27:50.469357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.463103 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w" event={"ID":"54d26225-aa3f-4a5f-a4f2-3c116cafe885","Type":"ContainerDied","Data":"8815fc01f213a107d840775e91fcee457c2743cd5d8e186f7d81a0c5d9f2aba2"} Oct 30 22:27:50.469649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.469628 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8815fc01f213a107d840775e91fcee457c2743cd5d8e186f7d81a0c5d9f2aba2" Oct 30 22:27:50.495300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.491597 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:50.495300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.491656 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:50.608354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.608317 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl"] Oct 30 22:27:50.620342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:50.620228 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b2e72-qs9rl"] Oct 30 22:27:51.498434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:51.498397 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:51.513580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:51.513535 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:52.192070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:52.192038 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66dd0b75-8823-4215-95ff-cd0a0116548c" path="/var/lib/kubelet/pods/66dd0b75-8823-4215-95ff-cd0a0116548c/volumes" Oct 30 22:27:52.485317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:52.485204 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:27:52.485471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:52.485315 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:27:53.490943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:53.490914 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:53.491439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:53.491416 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:53.492139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:53.492119 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:27:56.359288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:56.358373 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:27:57.002815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:57.002777 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:27:58.832629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:58.832577 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 22:27:58.833117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:27:58.832877 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" containerID="cri-o://0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820" gracePeriod=30 Oct 30 22:28:01.360015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.359970 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" podUID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" containerName="payload-processing" containerID="cri-o://fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8" gracePeriod=25 Oct 30 22:28:01.663683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.663656 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:28:01.682840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.682803 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\") pod \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " Oct 30 22:28:01.683088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.682895 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\") pod \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " Oct 30 22:28:01.683088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.683001 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\") pod \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " Oct 30 22:28:01.683088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.683052 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\") pod \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\" (UID: \"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb\") " Oct 30 22:28:01.684289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.683825 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume" pod "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" (UID: "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:01.684289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.684137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume" pod "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" (UID: "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:01.684289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.684190 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:01.686467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.686440 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb" pod "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" (UID: "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb"). InnerVolumeSpecName "kube-api-access-ksxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:01.688607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.688582 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials" pod "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" (UID: "a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:01.760917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.760878 5036 generic.go:356] "Generic (PLEG): container finished" podID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" containerID="fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8" exitCode=0 Oct 30 22:28:01.761112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.760969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" event={"ID":"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb","Type":"ContainerDied","Data":"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8"} Oct 30 22:28:01.761112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.761031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" event={"ID":"a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb","Type":"ContainerDied","Data":"c40f13ffecf3e702f0aa3f439920508360582f13b23400b0d5ca437452a355fb"} Oct 30 22:28:01.761112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.761054 5036 scope.go:122] "RemoveContainer" containerID="fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8" Oct 30 22:28:01.761112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.760981 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8" Oct 30 22:28:01.777445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.777419 5036 scope.go:122] "RemoveContainer" containerID="fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8" Oct 30 22:28:01.777752 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:01.777730 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8\": container with ID starting with fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8 not found: ID does not exist" containerID="fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8" Oct 30 22:28:01.777833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.777759 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8"} err="failed to get container status \"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8\": rpc error: code = NotFound desc = could not find container \"fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8\": container with ID starting with fa90dc72ba778dd5eb2d14cba12d9709bcbdf32762e850ed61810a9b573b51f8 not found: ID does not exist" Oct 30 22:28:01.785807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.785786 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:01.785807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.785808 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:01.785953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.785819 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxcb\" (UniqueName: \"kubernetes.io/projected/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb-kube-api-access-ksxcb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:01.789592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.789569 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:28:01.794549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:01.794530 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a39158/payload-processing-external-model-e2e-extmod-a-a39158-dd45jxsj8"] Oct 30 22:28:02.003789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.003750 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" podUID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" containerName="payload-processing" containerID="cri-o://73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0" gracePeriod=25 Oct 30 22:28:02.190652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.190540 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" path="/var/lib/kubelet/pods/a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb/volumes" Oct 30 22:28:02.283429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.283399 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:28:02.394478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.394450 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\") pod \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " Oct 30 22:28:02.394891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.394573 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\") pod \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " Oct 30 22:28:02.394891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.394611 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\") pod \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " Oct 30 22:28:02.394891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.394715 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\") pod \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\" (UID: \"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820\") " Oct 30 22:28:02.394891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.394854 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume" pod "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" (UID: "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:02.395132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.395110 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume" pod "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" (UID: "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:02.395504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.395482 5036 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-routing-overlay-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.395579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.395509 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.396918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.396896 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp" pod "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" (UID: "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820"). InnerVolumeSpecName "kube-api-access-g7zwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:02.397471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.397449 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials" pod "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" (UID: "e0ed73a5-8dbb-455b-9a2d-44bb0aa04820"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:02.497213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.497171 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zwp\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-kube-api-access-g7zwp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.497213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.497207 5036 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820-provider-credentials\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.772515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.772481 5036 generic.go:356] "Generic (PLEG): container finished" podID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" containerID="73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0" exitCode=0 Oct 30 22:28:02.772719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.772536 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" Oct 30 22:28:02.772719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.772573 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" event={"ID":"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820","Type":"ContainerDied","Data":"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0"} Oct 30 22:28:02.772719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.772627 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2" event={"ID":"e0ed73a5-8dbb-455b-9a2d-44bb0aa04820","Type":"ContainerDied","Data":"6ed7305786899cb0891d85436745af55d83e255d40b49e428e914534176401b8"} Oct 30 22:28:02.772719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.772653 5036 scope.go:122] "RemoveContainer" containerID="73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0" Oct 30 22:28:02.788119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.788103 5036 scope.go:122] "RemoveContainer" containerID="73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0" Oct 30 22:28:02.788469 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:02.788434 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0\": container with ID starting with 73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0 not found: ID does not exist" containerID="73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0" Oct 30 22:28:02.788540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.788479 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0"} err="failed to get container status \"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0\": rpc error: code = NotFound desc = could not find container \"73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0\": container with ID starting with 73be25effc40ba4993fb0d4be90bdb0f427b0758a4686bdd881361f499185df0 not found: ID does not exist" Oct 30 22:28:02.805187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.805165 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:28:02.812374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:02.812347 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-66b7wpnr2"] Oct 30 22:28:04.197146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.197105 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 22:28:04.201826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.201799 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" path="/var/lib/kubelet/pods/e0ed73a5-8dbb-455b-9a2d-44bb0aa04820/volumes" Oct 30 22:28:04.233868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.233829 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234022 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234076 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234117 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234212 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234316 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234346 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234548 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.234647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.234633 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\") pod \"e2faffd2-41a1-48b4-9967-421734cd1ee3\" (UID: \"e2faffd2-41a1-48b4-9967-421734cd1ee3\") " Oct 30 22:28:04.235101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235075 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.235306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235223 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.235390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235331 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.235484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235465 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.235545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235492 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.235545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235508 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.235660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.235572 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.236515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.236486 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:04.240302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.238450 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:04.240302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.239661 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "kube-api-access-q9r87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:04.241850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.241821 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:04.242315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.242292 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy" pod "e2faffd2-41a1-48b4-9967-421734cd1ee3" (UID: "e2faffd2-41a1-48b4-9967-421734cd1ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.337705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337663 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9r87\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-kube-api-access-q9r87\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.337705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337710 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.337705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337726 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.338059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337739 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.338059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337754 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2faffd2-41a1-48b4-9967-421734cd1ee3-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.338059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.337812 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2faffd2-41a1-48b4-9967-421734cd1ee3-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.797320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.797288 5036 generic.go:356] "Generic (PLEG): container finished" podID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerID="0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820" exitCode=0 Oct 30 22:28:04.797494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.797378 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" Oct 30 22:28:04.797494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.797408 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerDied","Data":"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820"} Oct 30 22:28:04.797494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.797462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh" event={"ID":"e2faffd2-41a1-48b4-9967-421734cd1ee3","Type":"ContainerDied","Data":"ea00a84b07959903ff7a15b09374542eccb419ad2bf80bdfd0738e14d6c35fc3"} Oct 30 22:28:04.797494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.797484 5036 scope.go:122] "RemoveContainer" containerID="0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820" Oct 30 22:28:04.814039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.814020 5036 scope.go:122] "RemoveContainer" containerID="362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.833767 5036 scope.go:122] "RemoveContainer" containerID="0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:04.834762 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820\": container with ID starting with 0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820 not found: ID does not exist" containerID="0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.834798 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820"} err="failed to get container status \"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820\": rpc error: code = NotFound desc = could not find container \"0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820\": container with ID starting with 0508cc141f29e74fdb5af81076377502c15d887b0ddeef2422fc58674f5ec820 not found: ID does not exist" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.834823 5036 scope.go:122] "RemoveContainer" containerID="362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:04.835159 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba\": container with ID starting with 362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba not found: ID does not exist" containerID="362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba" Oct 30 22:28:04.836279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.835190 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba"} err="failed to get container status \"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba\": rpc error: code = NotFound desc = could not find container \"362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba\": container with ID starting with 362327719a8e90e68a4027b2cd9e6f9d4a3cf3f63090e504a0469f7bca495aba not found: ID does not exist" Oct 30 22:28:04.838373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.838345 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 22:28:04.846514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.846487 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-zjtmh"] Oct 30 22:28:04.853003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.852978 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:28:04.853989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.853965 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.853990 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854011 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854019 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854034 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" containerName="payload-processing" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854041 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" containerName="payload-processing" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854065 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" containerName="payload-processing" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854072 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" containerName="payload-processing" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854090 5036 cpu_manager.go:414] "containerMap: removing container" podUID="54d26225-aa3f-4a5f-a4f2-3c116cafe885" containerName="revoke-keys" Oct 30 22:28:04.854112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854096 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d26225-aa3f-4a5f-a4f2-3c116cafe885" containerName="revoke-keys" Oct 30 22:28:04.854586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854348 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854366 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" containerName="istio-proxy" Oct 30 22:28:04.854586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854377 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0607d32-33bc-4933-9a8a-4f3cf3a0d8cb" containerName="payload-processing" Oct 30 22:28:04.854586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854392 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ed73a5-8dbb-455b-9a2d-44bb0aa04820" containerName="payload-processing" Oct 30 22:28:04.854586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.854404 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d26225-aa3f-4a5f-a4f2-3c116cafe885" containerName="revoke-keys" Oct 30 22:28:04.903177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.902337 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:28:04.903177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.902513 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.913959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.913787 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f60026-openshift-default-dockercfg-5slwr\"" Oct 30 22:28:04.950224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950233 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950440 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9xs\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950570 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950744 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:04.950911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:04.950897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.067348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.057532 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.067348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.057892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.067348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.057964 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.067348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.057996 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.067348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.058244 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.077288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.068350 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.077288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.068610 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.077288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.068725 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9xs\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.077288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.068821 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.079017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.078983 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.086289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.079472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.086489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.084944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.086625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.085180 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.086735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.085430 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.101287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.091607 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.101287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.098066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.113145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.113113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.136724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.136654 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9xs\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.223624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.223248 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:05.680912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.680870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:28:05.682370 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:28:05.682335 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17034993_a1cf_493a_a891_7e00d49f1a54.slice/crio-90ea8fa06f5315ad7ad82d828e585d9247242c08ba0e4e89449fb053fad76fad WatchSource:0}: Error finding container 90ea8fa06f5315ad7ad82d828e585d9247242c08ba0e4e89449fb053fad76fad: Status 404 returned error can't find the container with id 90ea8fa06f5315ad7ad82d828e585d9247242c08ba0e4e89449fb053fad76fad Oct 30 22:28:05.685692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.685659 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:05.685802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.685727 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:05.685802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.685759 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:05.814827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.814790 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerStarted","Data":"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c"} Oct 30 22:28:05.815050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.814852 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerStarted","Data":"90ea8fa06f5315ad7ad82d828e585d9247242c08ba0e4e89449fb053fad76fad"} Oct 30 22:28:05.840544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:05.840487 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podStartSLOduration=1.8404726839999999 podStartE2EDuration="1.840472684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:05.836834524 +0000 UTC m=+2909.786789471" watchObservedRunningTime="2026-10-30 22:28:05.840472684 +0000 UTC m=+2909.790427631" Oct 30 22:28:06.194607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:06.194574 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2faffd2-41a1-48b4-9967-421734cd1ee3" path="/var/lib/kubelet/pods/e2faffd2-41a1-48b4-9967-421734cd1ee3/volumes" Oct 30 22:28:06.224126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:06.224094 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:06.225625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:06.225595 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:28:06.225765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:06.225644 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:28:07.230172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:07.230141 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:07.853635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:07.852057 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:07.856430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:07.856076 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:09.476028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.475985 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:28:09.484111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.484074 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.487241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.487218 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:28:09.488527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.488494 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f60026\"" Oct 30 22:28:09.488689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.488590 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f60026-dockercfg-g4t78\"" Oct 30 22:28:09.494907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.494882 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:28:09.501162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.501142 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.522849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.522799 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:28:09.651677 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.651637 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvpv\" (UniqueName: \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.652051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.652027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.656933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.652248 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqt22\" (UniqueName: \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.676237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.676206 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.781311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.781194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvpv\" (UniqueName: \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.782080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.782045 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.782652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.782632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqt22\" (UniqueName: \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.783167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.783148 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.785109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.784180 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.785250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.784801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.810103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.809984 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvpv\" (UniqueName: \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\") pod \"payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:09.810929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.810907 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqt22\" (UniqueName: \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\") pod \"payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:09.839240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:09.839210 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:10.157303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.148627 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:10.205837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.205802 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:28:10.207227 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:28:10.207198 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c385c1_90d0_4da7_9c0a_012fa8fee7fd.slice/crio-652c7f91b7b18028d4f03ee775fa09a4bead33a2137cf9d1d5e69be021ee80bf WatchSource:0}: Error finding container 652c7f91b7b18028d4f03ee775fa09a4bead33a2137cf9d1d5e69be021ee80bf: Status 404 returned error can't find the container with id 652c7f91b7b18028d4f03ee775fa09a4bead33a2137cf9d1d5e69be021ee80bf Oct 30 22:28:10.478691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.478662 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:28:10.480699 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:28:10.480666 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051b4d4f_ad8c_4583_828c_74ebe8aba86f.slice/crio-9321beb38916d395a21cbc20eae0c960e670075a70824dcb81843011f96c53aa WatchSource:0}: Error finding container 9321beb38916d395a21cbc20eae0c960e670075a70824dcb81843011f96c53aa: Status 404 returned error can't find the container with id 9321beb38916d395a21cbc20eae0c960e670075a70824dcb81843011f96c53aa Oct 30 22:28:10.938445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.938397 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" event={"ID":"46c385c1-90d0-4da7-9c0a-012fa8fee7fd","Type":"ContainerStarted","Data":"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab"} Oct 30 22:28:10.938660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.938466 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" event={"ID":"46c385c1-90d0-4da7-9c0a-012fa8fee7fd","Type":"ContainerStarted","Data":"652c7f91b7b18028d4f03ee775fa09a4bead33a2137cf9d1d5e69be021ee80bf"} Oct 30 22:28:10.938660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.938499 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:10.940706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.940680 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" event={"ID":"051b4d4f-ad8c-4583-828c-74ebe8aba86f","Type":"ContainerStarted","Data":"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50"} Oct 30 22:28:10.940855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.940724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" event={"ID":"051b4d4f-ad8c-4583-828c-74ebe8aba86f","Type":"ContainerStarted","Data":"9321beb38916d395a21cbc20eae0c960e670075a70824dcb81843011f96c53aa"} Oct 30 22:28:10.940855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.940754 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:10.956031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.955981 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" podStartSLOduration=1.955965168 podStartE2EDuration="1.955965168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:10.955033872 +0000 UTC m=+2914.904988832" watchObservedRunningTime="2026-10-30 22:28:10.955965168 +0000 UTC m=+2914.905920117" Oct 30 22:28:10.991656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:10.991611 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" podStartSLOduration=1.991597936 podStartE2EDuration="1.991597936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:10.990241562 +0000 UTC m=+2914.940196519" watchObservedRunningTime="2026-10-30 22:28:10.991597936 +0000 UTC m=+2914.941552923" Oct 30 22:28:18.100683 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:18.100590 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:28:21.987287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:21.983449 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:28:21.994284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:21.988548 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:28:29.030434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:29.030358 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2"] Oct 30 22:28:29.036530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:29.036502 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961d28c-mh2x2"] Oct 30 22:28:30.190972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:30.190942 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e05283-26cb-4592-95aa-3586d17c19bf" path="/var/lib/kubelet/pods/50e05283-26cb-4592-95aa-3586d17c19bf/volumes" Oct 30 22:28:48.100720 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:28:48.100684 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:28:49.605713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.605683 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:28:49.667626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.667588 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.675748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.675710 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:28:49.676605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.676581 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-06ac2e-openshift-default-dockercfg-qdgwv\"" Oct 30 22:28:49.701937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.701905 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.701957 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702041 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702185 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702318 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702343 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:49.702558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:49.702369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz8m\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.006526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.006481 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.006958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.006932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.007158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.007140 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.007486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.007462 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.007657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.007636 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.007812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.007796 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.007969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.007951 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz8m\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.008180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.008158 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.008353 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.008334 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.056859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.036728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.056859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.054217 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.056859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.054575 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.056859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.055616 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.056859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.055901 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.206421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.206374 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.206612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.206461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.206776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.206749 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.239212 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.239168 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz8m\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\") pod \"e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.298286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.298249 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:50.666341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.664351 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:50.670322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.666968 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:28:50.670322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.669684 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:50.670322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:50.669760 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:51.201164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:51.201124 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerStarted","Data":"980ea0b135631f8ead6413179f1fbf50ab864b5317440acd590c999ba20dd928"} Oct 30 22:28:51.201414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:51.201190 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerStarted","Data":"dff16bd22487fccc052457714404c51ffec9cd95bf155003f23e9a74a4db9bda"} Oct 30 22:28:51.299334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:51.299275 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:51.307889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:51.307861 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:51.334635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:51.334573 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podStartSLOduration=2.334552377 podStartE2EDuration="2.334552377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.274775974 +0000 UTC m=+2955.224730948" watchObservedRunningTime="2026-10-30 22:28:51.334552377 +0000 UTC m=+2955.284507325" Oct 30 22:28:52.209771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:52.209744 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:52.210721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:52.210695 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:28:53.894434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:53.894394 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:28:53.894806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:53.894454 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:28:54.029320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.029277 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc"] Oct 30 22:28:54.034291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.034250 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c574e7-2jdrc"] Oct 30 22:28:54.191159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.191081 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d70e4d-8e79-4042-9bea-28a4f4fbdf1a" path="/var/lib/kubelet/pods/67d70e4d-8e79-4042-9bea-28a4f4fbdf1a/volumes" Oct 30 22:28:54.958204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.958169 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:28:54.964592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.964570 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:54.968283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.968241 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-06ac2e-dockercfg-6t9xn\"" Oct 30 22:28:54.970566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.970544 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-06ac2e\"" Oct 30 22:28:54.974435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:54.974411 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:28:55.021844 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.021812 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:28:55.029244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.028627 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.082293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.050393 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:28:55.343370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.314471 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmrjr\" (UniqueName: \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.343370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.314895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.426291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.423942 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7t9\" (UniqueName: \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.426291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.424060 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.426291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.424104 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.426291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.424306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmrjr\" (UniqueName: \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.426291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.425703 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.451354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.451255 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmrjr\" (UniqueName: \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\") pod \"payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.526561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.526525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7t9\" (UniqueName: \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.526835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.526668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.527506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.527481 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.539798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.539764 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7t9\" (UniqueName: \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\") pod \"payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.594622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.594555 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:55.655102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.655070 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:55.910058 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:28:55.910029 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02c42cb_3b4b_450a_941a_984a6b657906.slice/crio-ba0bded85ce28bfd5e50977dc119638dd7af1557a79d1df44654efa6e5fa5c00 WatchSource:0}: Error finding container ba0bded85ce28bfd5e50977dc119638dd7af1557a79d1df44654efa6e5fa5c00: Status 404 returned error can't find the container with id ba0bded85ce28bfd5e50977dc119638dd7af1557a79d1df44654efa6e5fa5c00 Oct 30 22:28:55.911824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:55.911782 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:28:56.091013 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:28:56.090982 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63377bef_457a_4395_9003_0c799b7c031b.slice/crio-d8e01ae84aa66fb8482f5f985dd81f805d8492bcd9d8d9aa4ce72d8195b1998a WatchSource:0}: Error finding container d8e01ae84aa66fb8482f5f985dd81f805d8492bcd9d8d9aa4ce72d8195b1998a: Status 404 returned error can't find the container with id d8e01ae84aa66fb8482f5f985dd81f805d8492bcd9d8d9aa4ce72d8195b1998a Oct 30 22:28:56.091392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.091133 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:28:56.463241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.463189 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" event={"ID":"63377bef-457a-4395-9003-0c799b7c031b","Type":"ContainerStarted","Data":"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939"} Oct 30 22:28:56.463489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.463256 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" event={"ID":"63377bef-457a-4395-9003-0c799b7c031b","Type":"ContainerStarted","Data":"d8e01ae84aa66fb8482f5f985dd81f805d8492bcd9d8d9aa4ce72d8195b1998a"} Oct 30 22:28:56.463489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.463344 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:28:56.465491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.465460 5036 generic.go:356] "Generic (PLEG): container finished" podID="17034993-a1cf-493a-a891-7e00d49f1a54" containerID="277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c" exitCode=0 Oct 30 22:28:56.465634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.465537 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerDied","Data":"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c"} Oct 30 22:28:56.466053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.466028 5036 scope.go:122] "RemoveContainer" containerID="277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c" Oct 30 22:28:56.468091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" event={"ID":"f02c42cb-3b4b-450a-941a-984a6b657906","Type":"ContainerStarted","Data":"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68"} Oct 30 22:28:56.468219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468111 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" event={"ID":"f02c42cb-3b4b-450a-941a-984a6b657906","Type":"ContainerStarted","Data":"ba0bded85ce28bfd5e50977dc119638dd7af1557a79d1df44654efa6e5fa5c00"} Oct 30 22:28:56.468219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:28:56.468443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468405 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:56.468524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468500 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:56.468587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.468554 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:56.501813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.501743 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" podStartSLOduration=2.488465964 podStartE2EDuration="2.488465964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.482001183 +0000 UTC m=+2960.431956130" watchObservedRunningTime="2026-10-30 22:28:56.488465964 +0000 UTC m=+2960.438420911" Oct 30 22:28:56.585935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:56.585874 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" podStartSLOduration=2.585855479 podStartE2EDuration="2.585855479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.567239936 +0000 UTC m=+2960.517194884" watchObservedRunningTime="2026-10-30 22:28:56.585855479 +0000 UTC m=+2960.535810425" Oct 30 22:28:57.499491 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:57.499450 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerStarted","Data":"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1"} Oct 30 22:28:58.152049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.152006 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:28:58.159968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.159933 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.163148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.163116 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:28:58.167226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167463 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167529 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167705 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167751 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.167892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.167873 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqp4\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.168025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.168000 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.248295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.240138 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:28:58.248295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.240354 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:28:58.248295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.240401 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:28:58.269602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.269569 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqp4\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.269901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.269873 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.269987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.269958 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270150 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270191 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270181 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270446 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.270746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.270649 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.271362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.271176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.271506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.271445 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.271607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.271581 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.271607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.271581 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.273503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.273432 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.273503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.273443 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.281916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.281889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqp4\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.282832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.282815 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.483324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.483223 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:28:58.860508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.860477 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:28:58.861025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.860982 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:58.861103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.861082 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:58.861160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:58.861141 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:59.244767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:59.244731 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:28:59.245069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:59.245039 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:28:59.550203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:59.550109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" event={"ID":"87b77dbe-fd9d-416d-a02f-66a8eaf39125","Type":"ContainerStarted","Data":"2ec1b170c6fd2a13a2efe178f45543f2e0d5044c131a199d57634cdde3b2b417"} Oct 30 22:28:59.550203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:59.550179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" event={"ID":"87b77dbe-fd9d-416d-a02f-66a8eaf39125","Type":"ContainerStarted","Data":"232ed8bce16bcd545452c136cd60889b1807817f41e94c17dcac32f5cb5f55a6"} Oct 30 22:28:59.577771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:28:59.577706 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podStartSLOduration=1.577685946 podStartE2EDuration="1.577685946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:59.573811327 +0000 UTC m=+2963.523766306" watchObservedRunningTime="2026-10-30 22:28:59.577685946 +0000 UTC m=+2963.527640893" Oct 30 22:29:00.227434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:00.227399 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.228628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:00.228593 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:29:00.484059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:00.483959 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:29:00.488339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:00.488303 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.488486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:00.488385 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:01.229290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:01.228756 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.229290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:01.228820 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:29:01.483813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:01.483717 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.483813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:01.483790 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:02.234294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:02.232788 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.234294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:02.232850 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:29:02.484099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:02.484060 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.484519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:02.484450 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:03.240291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:03.227674 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body= Oct 30 22:29:03.240291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:03.227740 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" Oct 30 22:29:03.484146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:03.484111 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:03.484476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:03.484177 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:04.248864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:04.246382 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:29:04.248864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:04.247227 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:29:04.248864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:04.247889 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:29:04.483854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:04.483812 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:04.484007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:04.483882 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:05.081334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:05.080326 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj"] Oct 30 22:29:05.099890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:05.099856 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c8d104-qphpj"] Oct 30 22:29:05.488338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:05.484259 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 30 22:29:05.488338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:05.484332 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 30 22:29:06.195010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:06.194976 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd46718-de10-45f2-b585-22e048672549" path="/var/lib/kubelet/pods/4bd46718-de10-45f2-b585-22e048672549/volumes" Oct 30 22:29:06.489283 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:06.489199 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:29:06.489719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:06.489703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:29:06.490479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:06.490457 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:29:07.503091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:07.503062 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:29:07.503618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:07.503143 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:29:14.036525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:14.036487 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh"] Oct 30 22:29:14.041782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:14.041753 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b878d9-9nggh"] Oct 30 22:29:14.210423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:14.210383 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0686b449-62f1-4d6c-8bba-9388c7cab3e2" path="/var/lib/kubelet/pods/0686b449-62f1-4d6c-8bba-9388c7cab3e2/volumes" Oct 30 22:29:15.033084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:15.033054 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2"] Oct 30 22:29:15.038190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:15.038159 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843f4177-cbld2"] Oct 30 22:29:16.195320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:16.195254 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7a22f8-f993-43bb-bbcc-1671af83a63e" path="/var/lib/kubelet/pods/dc7a22f8-f993-43bb-bbcc-1671af83a63e/volumes" Oct 30 22:29:18.101664 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:29:18.101625 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:29:24.028528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:24.028486 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528"] Oct 30 22:29:24.033411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:24.033384 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efd6f3e-wq528"] Oct 30 22:29:24.190826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:24.190795 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482b83d4-e366-46cb-a75d-6b0522dc607f" path="/var/lib/kubelet/pods/482b83d4-e366-46cb-a75d-6b0522dc607f/volumes" Oct 30 22:29:26.034581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:26.034539 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95"] Oct 30 22:29:26.042507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:26.042478 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5845f6-mnw95"] Oct 30 22:29:26.196518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:26.196483 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1990fc-e9c9-4ab2-b776-9a097950ce59" path="/var/lib/kubelet/pods/cd1990fc-e9c9-4ab2-b776-9a097950ce59/volumes" Oct 30 22:29:35.894637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.894592 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:29:35.900369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.900345 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.907735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.907711 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f60026\"/\"test-model-f60026-kserve-self-signed-certs\"" Oct 30 22:29:35.907946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.907927 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f60026\"/\"openshift-service-ca.crt\"" Oct 30 22:29:35.908041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.908022 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f60026\"/\"kube-root-ca.crt\"" Oct 30 22:29:35.908524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.908504 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f60026\"/\"default-dockercfg-6h9mk\"" Oct 30 22:29:35.911685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.911664 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:29:35.921830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.921802 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.923806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.923778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.924685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.924643 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.924933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.924904 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.925046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.924958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:35.925165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:35.925140 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gb2r\" (UniqueName: \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027105 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027133 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027185 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gb2r\" (UniqueName: \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027341 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027476 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027613 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.027977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.027904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.029689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.029665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.030369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.030349 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.039887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.039856 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gb2r\" (UniqueName: \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\") pod \"test-model-f60026-kserve-66b786576c-t68s8\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.234730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.234636 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:36.603152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:36.603121 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:29:36.603339 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:29:36.603160 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bcb57ce_0a29_4fc8_8002_3dc9606fe0ab.slice/crio-058642d5bf795cc79669feec4af5904ab6cbf83bf353290c9e60f500278e3e8f WatchSource:0}: Error finding container 058642d5bf795cc79669feec4af5904ab6cbf83bf353290c9e60f500278e3e8f: Status 404 returned error can't find the container with id 058642d5bf795cc79669feec4af5904ab6cbf83bf353290c9e60f500278e3e8f Oct 30 22:29:37.182893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.182854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" event={"ID":"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab","Type":"ContainerStarted","Data":"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552"} Oct 30 22:29:37.183399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.182909 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" event={"ID":"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab","Type":"ContainerStarted","Data":"058642d5bf795cc79669feec4af5904ab6cbf83bf353290c9e60f500278e3e8f"} Oct 30 22:29:37.206114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.206063 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" podStartSLOduration=2.206048988 podStartE2EDuration="2.206048988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:37.200162617 +0000 UTC m=+3001.150117563" watchObservedRunningTime="2026-10-30 22:29:37.206048988 +0000 UTC m=+3001.156003934" Oct 30 22:29:37.235526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.235499 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:37.298486 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.298458 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:29:37.299784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:37.299762 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:29:45.862354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.861551 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:29:45.867837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.867814 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:29:45.867998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.867925 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.872153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.872130 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-06ac2e\"/\"test-model-06ac2e-kserve-self-signed-certs\"" Oct 30 22:29:45.872620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.872603 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-06ac2e\"/\"kube-root-ca.crt\"" Oct 30 22:29:45.876915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.876897 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-06ac2e\"/\"default-dockercfg-4fscw\"" Oct 30 22:29:45.971578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.971541 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.971730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.971598 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.971794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.971730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.971968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.971941 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnsz\" (UniqueName: \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.972019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.971973 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-06ac2e\"/\"openshift-service-ca.crt\"" Oct 30 22:29:45.972111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.972096 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:45.972165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:45.972151 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081199 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081348 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnsz\" (UniqueName: \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081474 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081531 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.081787 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.082296 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.083580 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.083860 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.083844 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.117338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.102921 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnsz\" (UniqueName: \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.117338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.108556 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.165369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.154513 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\") pod \"test-model-06ac2e-kserve-746797cb77-m6v27\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.196646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.195989 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:46.292032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.292006 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:46.294995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.294971 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:46.381435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.381410 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:29:46.601028 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:29:46.600988 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d02fba_c5ff_4bfd_a97f_95d3b5d48197.slice/crio-1c18ff8edad63139e1f04f7e6005674531c8d9210a59566fd5d262932484858f WatchSource:0}: Error finding container 1c18ff8edad63139e1f04f7e6005674531c8d9210a59566fd5d262932484858f: Status 404 returned error can't find the container with id 1c18ff8edad63139e1f04f7e6005674531c8d9210a59566fd5d262932484858f Oct 30 22:29:46.601200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:46.601167 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:29:47.379123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:47.379081 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" event={"ID":"91d02fba-c5ff-4bfd-a97f-95d3b5d48197","Type":"ContainerStarted","Data":"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051"} Oct 30 22:29:47.379665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:47.379155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" event={"ID":"91d02fba-c5ff-4bfd-a97f-95d3b5d48197","Type":"ContainerStarted","Data":"1c18ff8edad63139e1f04f7e6005674531c8d9210a59566fd5d262932484858f"} Oct 30 22:29:47.481290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:47.480128 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" podStartSLOduration=2.480106432 podStartE2EDuration="2.480106432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:47.442009085 +0000 UTC m=+3011.391964032" watchObservedRunningTime="2026-10-30 22:29:47.480106432 +0000 UTC m=+3011.430061381" Oct 30 22:29:48.144348 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:29:48.137726 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:29:48.198397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:48.198365 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:50.298739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:50.298701 5036 scope.go:122] "RemoveContainer" containerID="a2aba00e7c37621986b03c8727642863720d4ed45f757a9e7d0a8a0eccc1525b" Oct 30 22:29:50.311130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:50.311101 5036 scope.go:122] "RemoveContainer" containerID="ec5f6348203296661616394277ea513a6e22d45a56c9c36e7cc74e5524f92d59" Oct 30 22:29:52.212237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.212165 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:52.212237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.212215 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:52.435175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.435139 5036 generic.go:356] "Generic (PLEG): container finished" podID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerID="980ea0b135631f8ead6413179f1fbf50ab864b5317440acd590c999ba20dd928" exitCode=0 Oct 30 22:29:52.435388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.435225 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerDied","Data":"980ea0b135631f8ead6413179f1fbf50ab864b5317440acd590c999ba20dd928"} Oct 30 22:29:52.435733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.435718 5036 scope.go:122] "RemoveContainer" containerID="980ea0b135631f8ead6413179f1fbf50ab864b5317440acd590c999ba20dd928" Oct 30 22:29:52.438133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.437887 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:52.438133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.438024 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:52.438133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:52.438065 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:53.447375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:53.447332 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerStarted","Data":"dff8110a3d121f5d4c0de089da739589cfe537546895ca4ab2accd7cdc3c4fc1"} Oct 30 22:29:54.088289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.087420 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd"] Oct 30 22:29:54.095295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.095073 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8"] Oct 30 22:29:54.101085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.101061 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8be69cb-nhwtd"] Oct 30 22:29:54.106340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.106316 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e813706-p9mm8"] Oct 30 22:29:54.191317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.191284 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedf517e-42fd-4ef3-b03d-9a097f301ed5" path="/var/lib/kubelet/pods/bedf517e-42fd-4ef3-b03d-9a097f301ed5/volumes" Oct 30 22:29:54.191830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.191814 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da44fe22-0efb-49d2-9309-6ef0577f03c1" path="/var/lib/kubelet/pods/da44fe22-0efb-49d2-9309-6ef0577f03c1/volumes" Oct 30 22:29:54.298912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.298877 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:29:54.299222 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.299187 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:54.299430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:54.299243 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:55.299333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:55.299256 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:55.299776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:55.299343 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:56.197564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:56.197530 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:56.322291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:56.300913 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:56.322915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:56.322874 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:56.357006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:56.356979 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:56.490013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:56.489921 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:29:57.299531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:57.299496 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:57.299721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:57.299546 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:58.305290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:58.304445 5036 patch_prober.go:29] interesting pod/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Oct 30 22:29:58.305290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:58.304508 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Oct 30 22:29:59.346606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:59.346567 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:29:59.347462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:59.347391 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:29:59.349134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:29:59.349110 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:30:00.159831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.159782 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:00.167133 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.167101 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:00.167344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.167207 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.174633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.174603 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:00.176703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.176673 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.177837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.177808 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.178061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.178041 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.178163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.178122 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:30:00.186973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.182910 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.201619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.201588 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:00.213124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.207675 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:00.213124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.207839 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.231495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.231434 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:00.231495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.231484 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:00.231725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.231585 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.238581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.238471 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:00.304581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.304506 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:00.372331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.372302 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:00.383210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.383184 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:00.389035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.388963 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.389254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wgc\" (UniqueName: \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.389458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389421 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdqb\" (UniqueName: \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.389458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389453 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.389594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389476 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdd8z\" (UniqueName: \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.389594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6hj\" (UniqueName: \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.389594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389575 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.389767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389652 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.389767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.389879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.389790 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkr5m\" (UniqueName: \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.390543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.390523 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.393466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.393257 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:00.401231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.401207 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:00.403544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.403516 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.407487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.407464 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.407824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.407800 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:00.418210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.418144 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.434546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.434512 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:00.442554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.442526 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:00.451071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.451046 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:00.460236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.460205 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:00.467553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.467526 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.474259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.474238 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:00.480630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.479858 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.491430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.491576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491466 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdqb\" (UniqueName: \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.491576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491505 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.491576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491531 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdd8z\" (UniqueName: \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.491576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491555 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpbv\" (UniqueName: \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.491576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6hj\" (UniqueName: \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491598 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491635 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491652 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kk5j\" (UniqueName: \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491823 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkr5m\" (UniqueName: \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.491877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rb8\" (UniqueName: \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491902 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbbs\" (UniqueName: \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.491998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.492018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wgc\" (UniqueName: \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.492048 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.492071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxw5\" (UniqueName: \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.492104 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.492521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.492153 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7rch\" (UniqueName: \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.493061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.493035 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.493613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.493591 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.494192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.494169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.494239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.494210 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.494975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.494949 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.495685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.495664 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:00.503575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.503559 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:00.504307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.503726 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.510241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.510198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wgc\" (UniqueName: \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\") pod \"maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.510412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.510318 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkr5m\" (UniqueName: \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\") pod \"maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.511311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.511185 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdqb\" (UniqueName: \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\") pod \"maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.519340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.511198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdd8z\" (UniqueName: \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\") pod \"maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.519340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.512704 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6hj\" (UniqueName: \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\") pod \"maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.526611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.521839 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:00.526611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.521970 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.547719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.547688 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:00.556159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.556131 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:00.562002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.561975 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.564037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.564015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:00.569503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.569481 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:00.574485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.574466 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:00.579804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.579779 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:00.585582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.585562 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:00.585716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.585697 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.592172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.592147 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:00.592294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.592252 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.593569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpbv\" (UniqueName: \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.593655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593588 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.593655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.593782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgvm\" (UniqueName: \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.593782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593691 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.593782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593751 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.593782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593775 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.594010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593864 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvhh\" (UniqueName: \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.594010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.593977 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.594109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594042 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kk5j\" (UniqueName: \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.594143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594096 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rb8\" (UniqueName: \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.594221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbbs\" (UniqueName: \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.594325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclz4\" (UniqueName: \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.594519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594467 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.594619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594561 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.594619 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.594747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594639 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxw5\" (UniqueName: \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.594747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.594869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7rch\" (UniqueName: \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.594869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594821 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.594995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.594866 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6p8w\" (UniqueName: \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.595325 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.595184 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.595872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.595833 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.595872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.595848 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.596039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.595952 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.596711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.596685 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.599649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.599629 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:00.599760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.599741 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.602762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.602741 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:00.609076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.609055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpbv\" (UniqueName: \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.609216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.609194 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kk5j\" (UniqueName: \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.611079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.611059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbbs\" (UniqueName: \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\") pod \"maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.611557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.611511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7rch\" (UniqueName: \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.611654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.611624 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rb8\" (UniqueName: \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\") pod \"maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.614323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.614298 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:00.617233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.617196 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:00.619104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.619080 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxw5\" (UniqueName: \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.625317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.625295 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:00.675130 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.675027 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:00.691595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.691563 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696369 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6p8w\" (UniqueName: \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696423 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcptg\" (UniqueName: \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696487 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696535 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgvm\" (UniqueName: \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696609 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696653 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvhh\" (UniqueName: \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.696811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696713 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrtg\" (UniqueName: \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.697613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.696967 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.697613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.697070 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclz4\" (UniqueName: \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.697613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.697189 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.697613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.697212 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.697613 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.697235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.697992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.697917 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.698060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.698003 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.698116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.698057 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.703654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.703505 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:00.712757 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.712694 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgvm\" (UniqueName: \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.714226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.714176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6p8w\" (UniqueName: \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.716029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.715999 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvhh\" (UniqueName: \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.717080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.717052 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclz4\" (UniqueName: \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.719014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.718821 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:00.745601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.745258 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:00.759875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.759667 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:00.768119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.767925 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:00.795994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.795955 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:00.802438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.802407 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcptg\" (UniqueName: \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.802579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.802558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrtg\" (UniqueName: \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.802830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.802773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.810527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.810486 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:00.813384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.813347 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.818348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.814288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.818348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.814982 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.818348 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.815219 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:00.825065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.824983 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcptg\" (UniqueName: \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:00.868568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.868297 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:00.875820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.875742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrtg\" (UniqueName: \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\") pod \"maas-api-key-cleanup-29889990-ddxws\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:00.877804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.877736 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:00.892574 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.892178 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:00.927095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.926795 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:00.927661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:00.927606 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:01.008992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.008965 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:01.182470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.180530 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:01.507754 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.507718 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:01.519060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.517512 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:01.522860 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:01.522418 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824b4b46_77b4_41a8_821f_8f0421ac3fab.slice/crio-3db168262723dd1d9643eec1144ee0c250253dc3e64b0d3d1d7ef9f515bdf360 WatchSource:0}: Error finding container 3db168262723dd1d9643eec1144ee0c250253dc3e64b0d3d1d7ef9f515bdf360: Status 404 returned error can't find the container with id 3db168262723dd1d9643eec1144ee0c250253dc3e64b0d3d1d7ef9f515bdf360 Oct 30 22:30:01.574307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.574020 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerStarted","Data":"3db168262723dd1d9643eec1144ee0c250253dc3e64b0d3d1d7ef9f515bdf360"} Oct 30 22:30:01.580862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.580826 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerStarted","Data":"b6a5a4befd1bd93e4f2a2a895a4a1a0ca2e084db9c0420ee05147b666249dd5e"} Oct 30 22:30:01.582998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.582964 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerStarted","Data":"b0b9e2eaef02eefdf7f716d4075060cea8847c5eb9aff4100badd2c385a61110"} Oct 30 22:30:01.690776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.690518 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:01.690776 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:01.690682 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd782b5c7_490f_4461_b0c9_8ac8a12e3462.slice/crio-1fbfbf85b0aed0c01c10f49b876b68fee2588247e060db3584e6977ac85a9128 WatchSource:0}: Error finding container 1fbfbf85b0aed0c01c10f49b876b68fee2588247e060db3584e6977ac85a9128: Status 404 returned error can't find the container with id 1fbfbf85b0aed0c01c10f49b876b68fee2588247e060db3584e6977ac85a9128 Oct 30 22:30:01.692059 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:01.692039 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b52730_29eb_49d4_8da6_9023f88646ad.slice/crio-2cf174f66ad08151087f19874a4e85395a4459777a18d79886abb9dffbbd7b25 WatchSource:0}: Error finding container 2cf174f66ad08151087f19874a4e85395a4459777a18d79886abb9dffbbd7b25: Status 404 returned error can't find the container with id 2cf174f66ad08151087f19874a4e85395a4459777a18d79886abb9dffbbd7b25 Oct 30 22:30:01.698071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.698046 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:01.730912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.730877 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:01.746925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.746898 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:01.785897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.785873 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:01.834073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.833615 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:01.862615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.856240 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:01.863477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.863453 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:01.871637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.871331 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:01.922819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.922790 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:01.997373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:01.997333 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:02.068646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.065322 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:02.106686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.106594 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:02.111010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.110967 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:02.140413 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:02.140391 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098d2d2e_595e_4d64_bd36_b9f72ad87d62.slice/crio-217c13d6c63e99d162737545b0908ecf6e47ff1e4c065e5bd3c8b8b2217ae939 WatchSource:0}: Error finding container 217c13d6c63e99d162737545b0908ecf6e47ff1e4c065e5bd3c8b8b2217ae939: Status 404 returned error can't find the container with id 217c13d6c63e99d162737545b0908ecf6e47ff1e4c065e5bd3c8b8b2217ae939 Oct 30 22:30:02.142933 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:02.142899 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5389037_db6e_4fa6_abfa_60c9e255cf77.slice/crio-3485a6e2bbac4b0fcdc5dbd09d4b5c89841c5ce1687ae7ec776492a26decce4f WatchSource:0}: Error finding container 3485a6e2bbac4b0fcdc5dbd09d4b5c89841c5ce1687ae7ec776492a26decce4f: Status 404 returned error can't find the container with id 3485a6e2bbac4b0fcdc5dbd09d4b5c89841c5ce1687ae7ec776492a26decce4f Oct 30 22:30:02.146999 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:02.146973 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa7c802_4abd_437b_abd0_2ab67b1f6f4d.slice/crio-ea4182654575d83767bc38522969ef107ceef6d89a50602b570bb8a35eebd103 WatchSource:0}: Error finding container ea4182654575d83767bc38522969ef107ceef6d89a50602b570bb8a35eebd103: Status 404 returned error can't find the container with id ea4182654575d83767bc38522969ef107ceef6d89a50602b570bb8a35eebd103 Oct 30 22:30:02.147368 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:02.147320 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718d3757_a41d_41ee_8d34_8e7d01022833.slice/crio-465b69140f4dec0073c3a6ecb82c7f010fa6d05c9ab2132d485c7f269b6a0f6f WatchSource:0}: Error finding container 465b69140f4dec0073c3a6ecb82c7f010fa6d05c9ab2132d485c7f269b6a0f6f: Status 404 returned error can't find the container with id 465b69140f4dec0073c3a6ecb82c7f010fa6d05c9ab2132d485c7f269b6a0f6f Oct 30 22:30:02.595753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.595660 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerStarted","Data":"13e7325faf92a66892ec4f7629904bb04674c96ef9f2ae9c26cc64d2f70cb1ed"} Oct 30 22:30:02.597186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.597158 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerStarted","Data":"7cebdd3679196c7b69d95181ae855a9563fc37bc976ca9c1393de572c31d736c"} Oct 30 22:30:02.598597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.598568 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" event={"ID":"d782b5c7-490f-4461-b0c9-8ac8a12e3462","Type":"ContainerStarted","Data":"1fbfbf85b0aed0c01c10f49b876b68fee2588247e060db3584e6977ac85a9128"} Oct 30 22:30:02.599905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.599878 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerStarted","Data":"f21c0338690a2aa1b6bc344a3106fd3ac2302ebbbaef7399708dcc7a0c94d718"} Oct 30 22:30:02.601807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.601781 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" event={"ID":"a5389037-db6e-4fa6-abfa-60c9e255cf77","Type":"ContainerStarted","Data":"3485a6e2bbac4b0fcdc5dbd09d4b5c89841c5ce1687ae7ec776492a26decce4f"} Oct 30 22:30:02.603049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.603026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" event={"ID":"e4b52730-29eb-49d4-8da6-9023f88646ad","Type":"ContainerStarted","Data":"2cf174f66ad08151087f19874a4e85395a4459777a18d79886abb9dffbbd7b25"} Oct 30 22:30:02.604759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.604735 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerStarted","Data":"5bc3472334db77a42fe60204a13fb84736fafd480fd8d9bf97e0b0be426bde93"} Oct 30 22:30:02.606881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.606856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerStarted","Data":"06802e8c49fe57fa3355d87e258515925e77b7b4390486fcdb3e12f081695bf4"} Oct 30 22:30:02.608631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.608607 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerStarted","Data":"9537c7dac034251a23d63650563c108b708dfc55146082e9588cf1659222d93e"} Oct 30 22:30:02.609983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.609961 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerStarted","Data":"217c13d6c63e99d162737545b0908ecf6e47ff1e4c065e5bd3c8b8b2217ae939"} Oct 30 22:30:02.611987 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.611960 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerStarted","Data":"234573849b93fcd2867cc9c0ffe4b34cc7007dc4ef5e05e06ddaf0919f613143"} Oct 30 22:30:02.617931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.617875 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" podStartSLOduration=2.617861596 podStartE2EDuration="2.617861596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.613595365 +0000 UTC m=+3026.563550312" watchObservedRunningTime="2026-10-30 22:30:02.617861596 +0000 UTC m=+3026.567816541" Oct 30 22:30:02.618499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.618471 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerStarted","Data":"c08baf81744c212fdc3849cd0b4d6b13dc18e28b6fb51dd17af38212e42fa23a"} Oct 30 22:30:02.620135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.620112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerStarted","Data":"ea4182654575d83767bc38522969ef107ceef6d89a50602b570bb8a35eebd103"} Oct 30 22:30:02.622363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.622318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerStarted","Data":"b1d3d902ef049e3139f7516830408808244adf9811092f8aea507eba38b887e1"} Oct 30 22:30:02.624680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.624656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerStarted","Data":"465b69140f4dec0073c3a6ecb82c7f010fa6d05c9ab2132d485c7f269b6a0f6f"} Oct 30 22:30:02.626951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.626928 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerStarted","Data":"954b7ff56a22c1ec9daba6a5d9853b18eef7b161bbb7f6ea26f276d5dd6fdea5"} Oct 30 22:30:02.629317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.629290 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerStarted","Data":"d1a9a0ca5c66337cc46d406924acd019581ece37547ba65b7c8322b509135dbb"} Oct 30 22:30:02.640901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.640853 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" podStartSLOduration=2.640841318 podStartE2EDuration="2.640841318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.634346687 +0000 UTC m=+3026.584301751" watchObservedRunningTime="2026-10-30 22:30:02.640841318 +0000 UTC m=+3026.590796264" Oct 30 22:30:02.665997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:02.665942 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" podStartSLOduration=2.665921731 podStartE2EDuration="2.665921731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.65083961 +0000 UTC m=+3026.600794579" watchObservedRunningTime="2026-10-30 22:30:02.665921731 +0000 UTC m=+3026.615876678" Oct 30 22:30:03.754311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.754123 5036 generic.go:356] "Generic (PLEG): container finished" podID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerID="9537c7dac034251a23d63650563c108b708dfc55146082e9588cf1659222d93e" exitCode=60 Oct 30 22:30:03.754311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.754247 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerDied","Data":"9537c7dac034251a23d63650563c108b708dfc55146082e9588cf1659222d93e"} Oct 30 22:30:03.754929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.754909 5036 scope.go:122] "RemoveContainer" containerID="9537c7dac034251a23d63650563c108b708dfc55146082e9588cf1659222d93e" Oct 30 22:30:03.757393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.756947 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerStarted","Data":"bb290d7b6f3aea57734a45003a58b6b6c12b572cc5cb5ac5126edf99e73816c7"} Oct 30 22:30:03.760965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.760939 5036 generic.go:356] "Generic (PLEG): container finished" podID="7e5a8505-b921-452e-8976-2921f116cbb2" containerID="8d9cf538f95980dbb58350bb47049ff34dc6bbacd2b80221fc293f9512ef2b1d" exitCode=60 Oct 30 22:30:03.761070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.761025 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerDied","Data":"8d9cf538f95980dbb58350bb47049ff34dc6bbacd2b80221fc293f9512ef2b1d"} Oct 30 22:30:03.761344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.761325 5036 scope.go:122] "RemoveContainer" containerID="8d9cf538f95980dbb58350bb47049ff34dc6bbacd2b80221fc293f9512ef2b1d" Oct 30 22:30:03.763717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.763694 5036 generic.go:356] "Generic (PLEG): container finished" podID="0530a85c-8a0f-417b-b430-92a298d192ab" containerID="c157c2a829a1c32a023c37a4479bb02b3667723c1b3bb25321f1d59177a4c874" exitCode=60 Oct 30 22:30:03.763806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.763771 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerDied","Data":"c157c2a829a1c32a023c37a4479bb02b3667723c1b3bb25321f1d59177a4c874"} Oct 30 22:30:03.764053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.764032 5036 scope.go:122] "RemoveContainer" containerID="c157c2a829a1c32a023c37a4479bb02b3667723c1b3bb25321f1d59177a4c874" Oct 30 22:30:03.766101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.766079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerStarted","Data":"f54d000337f69cb0b516c622b61e71f6e9f6a83c116882e08b5b393f0c63dde1"} Oct 30 22:30:03.767910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.767890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerStarted","Data":"5b66ec25b867e6eed7bca9532c5c6cabc8805c538caaecb8a242b7517a12592d"} Oct 30 22:30:03.768110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.768093 5036 scope.go:122] "RemoveContainer" containerID="5b66ec25b867e6eed7bca9532c5c6cabc8805c538caaecb8a242b7517a12592d" Oct 30 22:30:03.769698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.769676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerStarted","Data":"fadb2617c52327d560990948400ff7dbef737bd4ed735d6c16d12d23669f3862"} Oct 30 22:30:03.772169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.772135 5036 scope.go:122] "RemoveContainer" containerID="16357de87f26ce2b5e1edab7dd0f6a394a38a91e5633767a70212f9a3e2a8c14" Oct 30 22:30:03.772382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.772177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerStarted","Data":"16357de87f26ce2b5e1edab7dd0f6a394a38a91e5633767a70212f9a3e2a8c14"} Oct 30 22:30:03.776065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.776042 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerStarted","Data":"01a392719e9a3d380ce41789cf8d91eb756af2a00d022d9516629136e1915bb6"} Oct 30 22:30:03.776439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.776411 5036 scope.go:122] "RemoveContainer" containerID="01a392719e9a3d380ce41789cf8d91eb756af2a00d022d9516629136e1915bb6" Oct 30 22:30:03.779409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.779383 5036 generic.go:356] "Generic (PLEG): container finished" podID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerID="13e7325faf92a66892ec4f7629904bb04674c96ef9f2ae9c26cc64d2f70cb1ed" exitCode=60 Oct 30 22:30:03.779487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.779453 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerDied","Data":"13e7325faf92a66892ec4f7629904bb04674c96ef9f2ae9c26cc64d2f70cb1ed"} Oct 30 22:30:03.779749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.779731 5036 scope.go:122] "RemoveContainer" containerID="13e7325faf92a66892ec4f7629904bb04674c96ef9f2ae9c26cc64d2f70cb1ed" Oct 30 22:30:03.781407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.781385 5036 generic.go:356] "Generic (PLEG): container finished" podID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerID="54b33684a40868ad0815cf6368956d57ecd094ef0a7391da821f7cb2f1614374" exitCode=60 Oct 30 22:30:03.781489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.781462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerDied","Data":"54b33684a40868ad0815cf6368956d57ecd094ef0a7391da821f7cb2f1614374"} Oct 30 22:30:03.781670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.781648 5036 scope.go:122] "RemoveContainer" containerID="54b33684a40868ad0815cf6368956d57ecd094ef0a7391da821f7cb2f1614374" Oct 30 22:30:03.783367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.783347 5036 generic.go:356] "Generic (PLEG): container finished" podID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerID="ce4ec9af2726e396f4e7f9487e39862d3ba7c3f7be2390d3b6e09f99c7e2aca4" exitCode=60 Oct 30 22:30:03.783467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.783429 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" event={"ID":"d782b5c7-490f-4461-b0c9-8ac8a12e3462","Type":"ContainerDied","Data":"ce4ec9af2726e396f4e7f9487e39862d3ba7c3f7be2390d3b6e09f99c7e2aca4"} Oct 30 22:30:03.783632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.783615 5036 scope.go:122] "RemoveContainer" containerID="ce4ec9af2726e396f4e7f9487e39862d3ba7c3f7be2390d3b6e09f99c7e2aca4" Oct 30 22:30:03.785381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.785358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerStarted","Data":"1f94656a2ed84ed05ad6f52b499b6f1771145d55e9bf685a65bcd366f12f3422"} Oct 30 22:30:03.785641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.785621 5036 scope.go:122] "RemoveContainer" containerID="1f94656a2ed84ed05ad6f52b499b6f1771145d55e9bf685a65bcd366f12f3422" Oct 30 22:30:03.792515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.790109 5036 generic.go:356] "Generic (PLEG): container finished" podID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerID="930c51c4484edff294a137c0987f7be0737e97941263ad5919032f8530c1a692" exitCode=60 Oct 30 22:30:03.792515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.790247 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" event={"ID":"e4b52730-29eb-49d4-8da6-9023f88646ad","Type":"ContainerDied","Data":"930c51c4484edff294a137c0987f7be0737e97941263ad5919032f8530c1a692"} Oct 30 22:30:03.792515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.790438 5036 scope.go:122] "RemoveContainer" containerID="930c51c4484edff294a137c0987f7be0737e97941263ad5919032f8530c1a692" Oct 30 22:30:03.794742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.794715 5036 generic.go:356] "Generic (PLEG): container finished" podID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerID="5bc3472334db77a42fe60204a13fb84736fafd480fd8d9bf97e0b0be426bde93" exitCode=60 Oct 30 22:30:03.794876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.794793 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerDied","Data":"5bc3472334db77a42fe60204a13fb84736fafd480fd8d9bf97e0b0be426bde93"} Oct 30 22:30:03.796166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.795107 5036 scope.go:122] "RemoveContainer" containerID="5bc3472334db77a42fe60204a13fb84736fafd480fd8d9bf97e0b0be426bde93" Oct 30 22:30:03.807078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.807052 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerStarted","Data":"b5850f2587cd0003aed1636ee6934b925ca347d0646ce170ddfe036aedd4dcfe"} Oct 30 22:30:03.847161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.847105 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" podStartSLOduration=3.847085484 podStartE2EDuration="3.847085484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.844752572 +0000 UTC m=+3027.794707519" watchObservedRunningTime="2026-10-30 22:30:03.847085484 +0000 UTC m=+3027.797040430" Oct 30 22:30:03.875004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.874958 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" podStartSLOduration=3.874944332 podStartE2EDuration="3.874944332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.873183844 +0000 UTC m=+3027.823138791" watchObservedRunningTime="2026-10-30 22:30:03.874944332 +0000 UTC m=+3027.824899277" Oct 30 22:30:03.907155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:03.907110 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" podStartSLOduration=3.907095875 podStartE2EDuration="3.907095875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.901475255 +0000 UTC m=+3027.851430201" watchObservedRunningTime="2026-10-30 22:30:03.907095875 +0000 UTC m=+3027.857050897" Oct 30 22:30:04.039545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.039486 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" podStartSLOduration=4.039468085 podStartE2EDuration="4.039468085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.031306241 +0000 UTC m=+3027.981261187" watchObservedRunningTime="2026-10-30 22:30:04.039468085 +0000 UTC m=+3027.989423034" Oct 30 22:30:04.840249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.840206 5036 generic.go:356] "Generic (PLEG): container finished" podID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerID="bb290d7b6f3aea57734a45003a58b6b6c12b572cc5cb5ac5126edf99e73816c7" exitCode=60 Oct 30 22:30:04.840780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.840302 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerDied","Data":"bb290d7b6f3aea57734a45003a58b6b6c12b572cc5cb5ac5126edf99e73816c7"} Oct 30 22:30:04.840780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.840600 5036 scope.go:122] "RemoveContainer" containerID="bb290d7b6f3aea57734a45003a58b6b6c12b572cc5cb5ac5126edf99e73816c7" Oct 30 22:30:04.843565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.843541 5036 generic.go:356] "Generic (PLEG): container finished" podID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerID="f54d000337f69cb0b516c622b61e71f6e9f6a83c116882e08b5b393f0c63dde1" exitCode=60 Oct 30 22:30:04.843683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.843638 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerDied","Data":"f54d000337f69cb0b516c622b61e71f6e9f6a83c116882e08b5b393f0c63dde1"} Oct 30 22:30:04.844533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.844514 5036 scope.go:122] "RemoveContainer" containerID="f54d000337f69cb0b516c622b61e71f6e9f6a83c116882e08b5b393f0c63dde1" Oct 30 22:30:04.846127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.846107 5036 generic.go:356] "Generic (PLEG): container finished" podID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerID="5b66ec25b867e6eed7bca9532c5c6cabc8805c538caaecb8a242b7517a12592d" exitCode=60 Oct 30 22:30:04.846317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.846257 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerDied","Data":"5b66ec25b867e6eed7bca9532c5c6cabc8805c538caaecb8a242b7517a12592d"} Oct 30 22:30:04.848209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.848189 5036 generic.go:356] "Generic (PLEG): container finished" podID="718d3757-a41d-41ee-8d34-8e7d01022833" containerID="fadb2617c52327d560990948400ff7dbef737bd4ed735d6c16d12d23669f3862" exitCode=60 Oct 30 22:30:04.848326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.848278 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerDied","Data":"fadb2617c52327d560990948400ff7dbef737bd4ed735d6c16d12d23669f3862"} Oct 30 22:30:04.848540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.848525 5036 scope.go:122] "RemoveContainer" containerID="fadb2617c52327d560990948400ff7dbef737bd4ed735d6c16d12d23669f3862" Oct 30 22:30:04.850804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.850785 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerID="16357de87f26ce2b5e1edab7dd0f6a394a38a91e5633767a70212f9a3e2a8c14" exitCode=60 Oct 30 22:30:04.850918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.850849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerDied","Data":"16357de87f26ce2b5e1edab7dd0f6a394a38a91e5633767a70212f9a3e2a8c14"} Oct 30 22:30:04.852899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.852879 5036 generic.go:356] "Generic (PLEG): container finished" podID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerID="01a392719e9a3d380ce41789cf8d91eb756af2a00d022d9516629136e1915bb6" exitCode=60 Oct 30 22:30:04.852993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.852933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerDied","Data":"01a392719e9a3d380ce41789cf8d91eb756af2a00d022d9516629136e1915bb6"} Oct 30 22:30:04.855429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.855372 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerStarted","Data":"2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62"} Oct 30 22:30:04.859730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.859703 5036 generic.go:356] "Generic (PLEG): container finished" podID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerID="1f94656a2ed84ed05ad6f52b499b6f1771145d55e9bf685a65bcd366f12f3422" exitCode=60 Oct 30 22:30:04.859971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.859817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerDied","Data":"1f94656a2ed84ed05ad6f52b499b6f1771145d55e9bf685a65bcd366f12f3422"} Oct 30 22:30:04.861907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.861883 5036 generic.go:356] "Generic (PLEG): container finished" podID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerID="a0f3b281b61f433931152112b1c23de9deda61fd07cee4e4cf35b3dcb136fc6b" exitCode=60 Oct 30 22:30:04.862033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.861962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" event={"ID":"a5389037-db6e-4fa6-abfa-60c9e255cf77","Type":"ContainerDied","Data":"a0f3b281b61f433931152112b1c23de9deda61fd07cee4e4cf35b3dcb136fc6b"} Oct 30 22:30:04.862202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.862185 5036 scope.go:122] "RemoveContainer" containerID="a0f3b281b61f433931152112b1c23de9deda61fd07cee4e4cf35b3dcb136fc6b" Oct 30 22:30:04.863917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.863894 5036 generic.go:356] "Generic (PLEG): container finished" podID="72557df9-139d-4974-8363-ca23eed5fcb4" containerID="b5850f2587cd0003aed1636ee6934b925ca347d0646ce170ddfe036aedd4dcfe" exitCode=60 Oct 30 22:30:04.864009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.863973 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerDied","Data":"b5850f2587cd0003aed1636ee6934b925ca347d0646ce170ddfe036aedd4dcfe"} Oct 30 22:30:04.864145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:04.864130 5036 scope.go:122] "RemoveContainer" containerID="b5850f2587cd0003aed1636ee6934b925ca347d0646ce170ddfe036aedd4dcfe" Oct 30 22:30:05.880502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.880034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerStarted","Data":"20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0"} Oct 30 22:30:05.883024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.883000 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerID="c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834" exitCode=60 Oct 30 22:30:05.883143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.883085 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerDied","Data":"c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834"} Oct 30 22:30:05.883219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.883154 5036 scope.go:122] "RemoveContainer" containerID="16357de87f26ce2b5e1edab7dd0f6a394a38a91e5633767a70212f9a3e2a8c14" Oct 30 22:30:05.883474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.883458 5036 scope.go:122] "RemoveContainer" containerID="c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834" Oct 30 22:30:05.883874 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.883848 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj_odh-ai-gateway-infra(1d3a9c92-5817-480c-9a5d-e024e6fec13f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" Oct 30 22:30:05.885585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.885562 5036 generic.go:356] "Generic (PLEG): container finished" podID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerID="1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9" exitCode=60 Oct 30 22:30:05.885692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.885605 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerDied","Data":"1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9"} Oct 30 22:30:05.885885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.885870 5036 scope.go:122] "RemoveContainer" containerID="1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9" Oct 30 22:30:05.886209 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.886155 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx_odh-ai-gateway-infra(9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" Oct 30 22:30:05.887815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.887795 5036 generic.go:356] "Generic (PLEG): container finished" podID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerID="2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62" exitCode=60 Oct 30 22:30:05.887905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.887866 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerDied","Data":"2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62"} Oct 30 22:30:05.888092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.888075 5036 scope.go:122] "RemoveContainer" containerID="2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62" Oct 30 22:30:05.888386 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.888368 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96_odh-ai-gateway-infra(824b4b46-77b4-41a8-821f-8f0421ac3fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" Oct 30 22:30:05.890189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.890169 5036 generic.go:356] "Generic (PLEG): container finished" podID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerID="66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e" exitCode=60 Oct 30 22:30:05.890287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.890241 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerDied","Data":"66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e"} Oct 30 22:30:05.890555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.890536 5036 scope.go:122] "RemoveContainer" containerID="66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e" Oct 30 22:30:05.890893 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.890868 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8_odh-ai-gateway-infra(bc84c7b4-6dc0-41d4-bd55-c807fd444c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" Oct 30 22:30:05.892628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.892604 5036 generic.go:356] "Generic (PLEG): container finished" podID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerID="52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe" exitCode=60 Oct 30 22:30:05.892724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.892685 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" event={"ID":"d782b5c7-490f-4461-b0c9-8ac8a12e3462","Type":"ContainerDied","Data":"52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe"} Oct 30 22:30:05.892958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.892941 5036 scope.go:122] "RemoveContainer" containerID="52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe" Oct 30 22:30:05.893199 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.893176 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95_odh-ai-gateway-infra(d782b5c7-490f-4461-b0c9-8ac8a12e3462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" Oct 30 22:30:05.894991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.894971 5036 generic.go:356] "Generic (PLEG): container finished" podID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerID="4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301" exitCode=60 Oct 30 22:30:05.895091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.895030 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerDied","Data":"4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301"} Oct 30 22:30:05.895314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.895297 5036 scope.go:122] "RemoveContainer" containerID="4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301" Oct 30 22:30:05.895569 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.895548 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5_odh-ai-gateway-infra(1dd0862d-5888-4022-9d6b-f4bf203c45ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" Oct 30 22:30:05.897175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.897153 5036 generic.go:356] "Generic (PLEG): container finished" podID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerID="fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394" exitCode=60 Oct 30 22:30:05.897299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.897187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" event={"ID":"e4b52730-29eb-49d4-8da6-9023f88646ad","Type":"ContainerDied","Data":"fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394"} Oct 30 22:30:05.897566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.897545 5036 scope.go:122] "RemoveContainer" containerID="fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394" Oct 30 22:30:05.897828 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.897806 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj_odh-ai-gateway-infra(e4b52730-29eb-49d4-8da6-9023f88646ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" Oct 30 22:30:05.900407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.900383 5036 generic.go:356] "Generic (PLEG): container finished" podID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerID="14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce" exitCode=60 Oct 30 22:30:05.900525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.900419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerDied","Data":"14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce"} Oct 30 22:30:05.900741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.900721 5036 scope.go:122] "RemoveContainer" containerID="14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce" Oct 30 22:30:05.901044 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.901006 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t_odh-ai-gateway-infra(ed331f16-3f65-41f7-9907-50008df1ba4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" Oct 30 22:30:05.902580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.902560 5036 generic.go:356] "Generic (PLEG): container finished" podID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerID="0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca" exitCode=60 Oct 30 22:30:05.902673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.902636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerDied","Data":"0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca"} Oct 30 22:30:05.902862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.902847 5036 scope.go:122] "RemoveContainer" containerID="0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca" Oct 30 22:30:05.903090 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.903069 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j_odh-ai-gateway-infra(4b9090da-d338-4ba0-ada0-ea88934aacc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" Oct 30 22:30:05.904719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.904695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerStarted","Data":"42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556"} Oct 30 22:30:05.906718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.906682 5036 generic.go:356] "Generic (PLEG): container finished" podID="7e5a8505-b921-452e-8976-2921f116cbb2" containerID="cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939" exitCode=60 Oct 30 22:30:05.906803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.906756 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerDied","Data":"cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939"} Oct 30 22:30:05.906953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.906937 5036 scope.go:122] "RemoveContainer" containerID="cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939" Oct 30 22:30:05.907177 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.907154 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf_odh-ai-gateway-infra(7e5a8505-b921-452e-8976-2921f116cbb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" Oct 30 22:30:05.909069 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.909049 5036 generic.go:356] "Generic (PLEG): container finished" podID="0530a85c-8a0f-417b-b430-92a298d192ab" containerID="943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844" exitCode=60 Oct 30 22:30:05.909160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.909108 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerDied","Data":"943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844"} Oct 30 22:30:05.909469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.909449 5036 scope.go:122] "RemoveContainer" containerID="943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844" Oct 30 22:30:05.909752 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.909731 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-ddxws_odh-ai-gateway-infra(0530a85c-8a0f-417b-b430-92a298d192ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" Oct 30 22:30:05.919807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.914097 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerStarted","Data":"e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523"} Oct 30 22:30:05.919807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.917023 5036 generic.go:356] "Generic (PLEG): container finished" podID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerID="e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d" exitCode=60 Oct 30 22:30:05.919807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.917107 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerDied","Data":"e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d"} Oct 30 22:30:05.919807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:05.917782 5036 scope.go:122] "RemoveContainer" containerID="e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d" Oct 30 22:30:05.919807 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:05.918117 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7_odh-ai-gateway-infra(a2fe75d3-cc21-4e33-96aa-5918f6c6260a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" Oct 30 22:30:06.385316 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:06.384868 5036 scope.go:122] "RemoveContainer" containerID="01a392719e9a3d380ce41789cf8d91eb756af2a00d022d9516629136e1915bb6" Oct 30 22:30:06.526649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:06.526557 5036 scope.go:122] "RemoveContainer" containerID="13e7325faf92a66892ec4f7629904bb04674c96ef9f2ae9c26cc64d2f70cb1ed" Oct 30 22:30:06.849767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:06.845623 5036 scope.go:122] "RemoveContainer" containerID="54b33684a40868ad0815cf6368956d57ecd094ef0a7391da821f7cb2f1614374" Oct 30 22:30:06.869730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:06.869706 5036 scope.go:122] "RemoveContainer" containerID="ce4ec9af2726e396f4e7f9487e39862d3ba7c3f7be2390d3b6e09f99c7e2aca4" Oct 30 22:30:06.979741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:06.961651 5036 scope.go:122] "RemoveContainer" containerID="1f94656a2ed84ed05ad6f52b499b6f1771145d55e9bf685a65bcd366f12f3422" Oct 30 22:30:07.136292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.134322 5036 generic.go:356] "Generic (PLEG): container finished" podID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerID="42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556" exitCode=60 Oct 30 22:30:07.136292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.134430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerDied","Data":"42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556"} Oct 30 22:30:07.136292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.134957 5036 scope.go:122] "RemoveContainer" containerID="42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556" Oct 30 22:30:07.136292 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.135450 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc_odh-ai-gateway-infra(098d2d2e-595e-4d64-bd36-b9f72ad87d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" Oct 30 22:30:07.177590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.170739 5036 generic.go:356] "Generic (PLEG): container finished" podID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerID="e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523" exitCode=60 Oct 30 22:30:07.177590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.170905 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerDied","Data":"e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523"} Oct 30 22:30:07.177590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.171442 5036 scope.go:122] "RemoveContainer" containerID="e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523" Oct 30 22:30:07.177590 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.171911 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp_odh-ai-gateway-infra(5aa7c802-4abd-437b-abd0-2ab67b1f6f4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" Oct 30 22:30:07.193326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.193301 5036 scope.go:122] "RemoveContainer" containerID="930c51c4484edff294a137c0987f7be0737e97941263ad5919032f8530c1a692" Oct 30 22:30:07.196500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.196473 5036 generic.go:356] "Generic (PLEG): container finished" podID="718d3757-a41d-41ee-8d34-8e7d01022833" containerID="20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0" exitCode=60 Oct 30 22:30:07.196770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.196745 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerDied","Data":"20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0"} Oct 30 22:30:07.197286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.197244 5036 scope.go:122] "RemoveContainer" containerID="20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0" Oct 30 22:30:07.204858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.204838 5036 scope.go:122] "RemoveContainer" containerID="c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834" Oct 30 22:30:07.205415 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.205338 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj_odh-ai-gateway-infra(1d3a9c92-5817-480c-9a5d-e024e6fec13f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" Oct 30 22:30:07.212384 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.212351 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8_odh-ai-gateway-infra(718d3757-a41d-41ee-8d34-8e7d01022833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" Oct 30 22:30:07.221695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.221387 5036 scope.go:122] "RemoveContainer" containerID="2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62" Oct 30 22:30:07.221695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.221538 5036 scope.go:122] "RemoveContainer" containerID="1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9" Oct 30 22:30:07.222689 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.221872 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx_odh-ai-gateway-infra(9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" Oct 30 22:30:07.222689 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.222637 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96_odh-ai-gateway-infra(824b4b46-77b4-41a8-821f-8f0421ac3fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" Oct 30 22:30:07.225887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.225867 5036 scope.go:122] "RemoveContainer" containerID="66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e" Oct 30 22:30:07.226356 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.226332 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8_odh-ai-gateway-infra(bc84c7b4-6dc0-41d4-bd55-c807fd444c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" Oct 30 22:30:07.227891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.227848 5036 scope.go:122] "RemoveContainer" containerID="52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe" Oct 30 22:30:07.228149 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.228127 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95_odh-ai-gateway-infra(d782b5c7-490f-4461-b0c9-8ac8a12e3462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" Oct 30 22:30:07.228347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.228328 5036 scope.go:122] "RemoveContainer" containerID="5bc3472334db77a42fe60204a13fb84736fafd480fd8d9bf97e0b0be426bde93" Oct 30 22:30:07.233647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.233621 5036 generic.go:356] "Generic (PLEG): container finished" podID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerID="0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac" exitCode=60 Oct 30 22:30:07.233817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.233798 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" event={"ID":"a5389037-db6e-4fa6-abfa-60c9e255cf77","Type":"ContainerDied","Data":"0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac"} Oct 30 22:30:07.234185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.234160 5036 scope.go:122] "RemoveContainer" containerID="0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac" Oct 30 22:30:07.234591 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.234562 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k_odh-ai-gateway-infra(a5389037-db6e-4fa6-abfa-60c9e255cf77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" Oct 30 22:30:07.237253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.237226 5036 generic.go:356] "Generic (PLEG): container finished" podID="72557df9-139d-4974-8363-ca23eed5fcb4" containerID="0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5" exitCode=60 Oct 30 22:30:07.237734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.237705 5036 scope.go:122] "RemoveContainer" containerID="0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5" Oct 30 22:30:07.238100 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.238075 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25_odh-ai-gateway-infra(72557df9-139d-4974-8363-ca23eed5fcb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" Oct 30 22:30:07.238645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.238626 5036 scope.go:122] "RemoveContainer" containerID="cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939" Oct 30 22:30:07.238845 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.238825 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf_odh-ai-gateway-infra(7e5a8505-b921-452e-8976-2921f116cbb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" Oct 30 22:30:07.238926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.238861 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerDied","Data":"0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5"} Oct 30 22:30:07.239118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.239101 5036 scope.go:122] "RemoveContainer" containerID="fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394" Oct 30 22:30:07.239354 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.239314 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj_odh-ai-gateway-infra(e4b52730-29eb-49d4-8da6-9023f88646ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" Oct 30 22:30:07.239718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.239673 5036 scope.go:122] "RemoveContainer" containerID="14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce" Oct 30 22:30:07.239978 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.239943 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t_odh-ai-gateway-infra(ed331f16-3f65-41f7-9907-50008df1ba4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" Oct 30 22:30:07.241033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.240314 5036 scope.go:122] "RemoveContainer" containerID="4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301" Oct 30 22:30:07.241033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.240429 5036 scope.go:122] "RemoveContainer" containerID="943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844" Oct 30 22:30:07.241033 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.240586 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5_odh-ai-gateway-infra(1dd0862d-5888-4022-9d6b-f4bf203c45ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" Oct 30 22:30:07.241033 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.240651 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-ddxws_odh-ai-gateway-infra(0530a85c-8a0f-417b-b430-92a298d192ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" Oct 30 22:30:07.241033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.240879 5036 scope.go:122] "RemoveContainer" containerID="e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d" Oct 30 22:30:07.241387 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.241109 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7_odh-ai-gateway-infra(a2fe75d3-cc21-4e33-96aa-5918f6c6260a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" Oct 30 22:30:07.241387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.241212 5036 scope.go:122] "RemoveContainer" containerID="0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca" Oct 30 22:30:07.241494 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:07.241441 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j_odh-ai-gateway-infra(4b9090da-d338-4ba0-ada0-ea88934aacc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" Oct 30 22:30:07.250749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.250732 5036 scope.go:122] "RemoveContainer" containerID="9537c7dac034251a23d63650563c108b708dfc55146082e9588cf1659222d93e" Oct 30 22:30:07.273150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.273121 5036 scope.go:122] "RemoveContainer" containerID="8d9cf538f95980dbb58350bb47049ff34dc6bbacd2b80221fc293f9512ef2b1d" Oct 30 22:30:07.295916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.295819 5036 scope.go:122] "RemoveContainer" containerID="c157c2a829a1c32a023c37a4479bb02b3667723c1b3bb25321f1d59177a4c874" Oct 30 22:30:07.317483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.317460 5036 scope.go:122] "RemoveContainer" containerID="5b66ec25b867e6eed7bca9532c5c6cabc8805c538caaecb8a242b7517a12592d" Oct 30 22:30:07.336227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.336188 5036 scope.go:122] "RemoveContainer" containerID="bb290d7b6f3aea57734a45003a58b6b6c12b572cc5cb5ac5126edf99e73816c7" Oct 30 22:30:07.434158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.434137 5036 scope.go:122] "RemoveContainer" containerID="f54d000337f69cb0b516c622b61e71f6e9f6a83c116882e08b5b393f0c63dde1" Oct 30 22:30:07.455903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.455867 5036 scope.go:122] "RemoveContainer" containerID="fadb2617c52327d560990948400ff7dbef737bd4ed735d6c16d12d23669f3862" Oct 30 22:30:07.471816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.471791 5036 scope.go:122] "RemoveContainer" containerID="a0f3b281b61f433931152112b1c23de9deda61fd07cee4e4cf35b3dcb136fc6b" Oct 30 22:30:07.487898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:07.487873 5036 scope.go:122] "RemoveContainer" containerID="b5850f2587cd0003aed1636ee6934b925ca347d0646ce170ddfe036aedd4dcfe" Oct 30 22:30:08.293971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:08.293939 5036 scope.go:122] "RemoveContainer" containerID="0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5" Oct 30 22:30:08.294488 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:08.294440 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25_odh-ai-gateway-infra(72557df9-139d-4974-8363-ca23eed5fcb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" Oct 30 22:30:08.297659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:08.297635 5036 scope.go:122] "RemoveContainer" containerID="42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556" Oct 30 22:30:08.298043 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:08.298017 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc_odh-ai-gateway-infra(098d2d2e-595e-4d64-bd36-b9f72ad87d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" Oct 30 22:30:08.302944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:08.302926 5036 scope.go:122] "RemoveContainer" containerID="e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523" Oct 30 22:30:08.303359 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:08.303332 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp_odh-ai-gateway-infra(5aa7c802-4abd-437b-abd0-2ab67b1f6f4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" Oct 30 22:30:08.306369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:08.306348 5036 scope.go:122] "RemoveContainer" containerID="20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0" Oct 30 22:30:08.306688 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:08.306662 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8_odh-ai-gateway-infra(718d3757-a41d-41ee-8d34-8e7d01022833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" Oct 30 22:30:08.309721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:08.309703 5036 scope.go:122] "RemoveContainer" containerID="0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac" Oct 30 22:30:08.309967 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:08.309949 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k_odh-ai-gateway-infra(a5389037-db6e-4fa6-abfa-60c9e255cf77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" Oct 30 22:30:17.180861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:17.180833 5036 scope.go:122] "RemoveContainer" containerID="1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9" Oct 30 22:30:17.181372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:17.180969 5036 scope.go:122] "RemoveContainer" containerID="20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0" Oct 30 22:30:18.101257 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:18.101211 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:30:18.180343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.180311 5036 scope.go:122] "RemoveContainer" containerID="cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939" Oct 30 22:30:18.180572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.180553 5036 scope.go:122] "RemoveContainer" containerID="fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394" Oct 30 22:30:18.180711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.180690 5036 scope.go:122] "RemoveContainer" containerID="42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556" Oct 30 22:30:18.181259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.181238 5036 scope.go:122] "RemoveContainer" containerID="4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301" Oct 30 22:30:18.183093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.182473 5036 scope.go:122] "RemoveContainer" containerID="943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844" Oct 30 22:30:18.588372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.588337 5036 generic.go:356] "Generic (PLEG): container finished" podID="718d3757-a41d-41ee-8d34-8e7d01022833" containerID="731977ff93e8f20f11c09ac52727d574804477fca1229a70ce8a9a6378a694ef" exitCode=60 Oct 30 22:30:18.588514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.588410 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerDied","Data":"731977ff93e8f20f11c09ac52727d574804477fca1229a70ce8a9a6378a694ef"} Oct 30 22:30:18.588514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.588494 5036 scope.go:122] "RemoveContainer" containerID="20e134052035c84d2c9259b685a3eceee4c01f6408c6fd13b6bd6c0e405099e0" Oct 30 22:30:18.588958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.588936 5036 scope.go:122] "RemoveContainer" containerID="731977ff93e8f20f11c09ac52727d574804477fca1229a70ce8a9a6378a694ef" Oct 30 22:30:18.589504 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:18.589475 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8_odh-ai-gateway-infra(718d3757-a41d-41ee-8d34-8e7d01022833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" Oct 30 22:30:18.592054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.591885 5036 generic.go:356] "Generic (PLEG): container finished" podID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerID="efaa9bf971608cf3b2516c184312080f2bbca13d103b698608afabfbdc89d0c9" exitCode=60 Oct 30 22:30:18.592158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.592067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerDied","Data":"efaa9bf971608cf3b2516c184312080f2bbca13d103b698608afabfbdc89d0c9"} Oct 30 22:30:18.592638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.592475 5036 scope.go:122] "RemoveContainer" containerID="efaa9bf971608cf3b2516c184312080f2bbca13d103b698608afabfbdc89d0c9" Oct 30 22:30:18.593224 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:18.593197 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx_odh-ai-gateway-infra(9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" Oct 30 22:30:18.684482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:18.684459 5036 scope.go:122] "RemoveContainer" containerID="1d362b86265e7b369fdd7d8e62e37e3b4af0c4b6a5e14062e03448da9506cdd9" Oct 30 22:30:19.184470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.184425 5036 scope.go:122] "RemoveContainer" containerID="2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62" Oct 30 22:30:19.186381 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.186356 5036 scope.go:122] "RemoveContainer" containerID="0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca" Oct 30 22:30:19.197869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.194772 5036 scope.go:122] "RemoveContainer" containerID="e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d" Oct 30 22:30:19.197869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.194876 5036 scope.go:122] "RemoveContainer" containerID="66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e" Oct 30 22:30:19.605658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.604422 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerStarted","Data":"b4b0a8a741cca81a4246f2d722b6a698f76ea4103c053227c4a800704f3e4d6a"} Oct 30 22:30:19.612964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.611255 5036 generic.go:356] "Generic (PLEG): container finished" podID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerID="45d2ad9a2a370ada4c743aec3edcf386140bbf509dac6f400209dc91a15a16c5" exitCode=60 Oct 30 22:30:19.612964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.611520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" event={"ID":"e4b52730-29eb-49d4-8da6-9023f88646ad","Type":"ContainerDied","Data":"45d2ad9a2a370ada4c743aec3edcf386140bbf509dac6f400209dc91a15a16c5"} Oct 30 22:30:19.612964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.611585 5036 scope.go:122] "RemoveContainer" containerID="fca9a46734d0e778abf28556528fb99f32c57a1cfe447040970373d9dbf70394" Oct 30 22:30:19.612964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.612083 5036 scope.go:122] "RemoveContainer" containerID="45d2ad9a2a370ada4c743aec3edcf386140bbf509dac6f400209dc91a15a16c5" Oct 30 22:30:19.612964 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:19.612572 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj_odh-ai-gateway-infra(e4b52730-29eb-49d4-8da6-9023f88646ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" Oct 30 22:30:19.622155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.621206 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerStarted","Data":"3620f1747130ed1759e2eba666c43d8b5f45f2f33ccf92f1cbe357bc2510b8f2"} Oct 30 22:30:19.625698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.625666 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerStarted","Data":"71dadbd0795da56847cfb34cd0183f30b469bd1db9a690e8726f9857d959786e"} Oct 30 22:30:19.628009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.627985 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerStarted","Data":"e76aeae4b8fd6ce75e90329526485a97a60c8f700df877b07193e9722a61cf1a"} Oct 30 22:30:19.629333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.629306 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:19.639062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.639021 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" podStartSLOduration=19.639008404 podStartE2EDuration="19.639008404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:19.63256904 +0000 UTC m=+3043.582523987" watchObservedRunningTime="2026-10-30 22:30:19.639008404 +0000 UTC m=+3043.588963350" Oct 30 22:30:19.647313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.647249 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:19.664415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.664366 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" podStartSLOduration=19.664352378 podStartE2EDuration="19.664352378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:19.651634432 +0000 UTC m=+3043.601589378" watchObservedRunningTime="2026-10-30 22:30:19.664352378 +0000 UTC m=+3043.614307325" Oct 30 22:30:19.717031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.716903 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" podStartSLOduration=19.716884 podStartE2EDuration="19.716884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:19.713515775 +0000 UTC m=+3043.663470722" watchObservedRunningTime="2026-10-30 22:30:19.716884 +0000 UTC m=+3043.666838948" Oct 30 22:30:19.878907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.878883 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:19.967549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.967515 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\" (UniqueName: \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\") pod \"718d3757-a41d-41ee-8d34-8e7d01022833\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " Oct 30 22:30:19.969288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.969241 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\") pod \"718d3757-a41d-41ee-8d34-8e7d01022833\" (UID: \"718d3757-a41d-41ee-8d34-8e7d01022833\") " Oct 30 22:30:19.971223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:19.971195 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca" pod "718d3757-a41d-41ee-8d34-8e7d01022833" (UID: "718d3757-a41d-41ee-8d34-8e7d01022833"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.022941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.022871 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm" pod "718d3757-a41d-41ee-8d34-8e7d01022833" (UID: "718d3757-a41d-41ee-8d34-8e7d01022833"). InnerVolumeSpecName "kube-api-access-8lgvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.051418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.051390 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:20.072831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.072705 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgvm\" (UniqueName: \"kubernetes.io/projected/718d3757-a41d-41ee-8d34-8e7d01022833-kube-api-access-8lgvm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.072831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.072734 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/718d3757-a41d-41ee-8d34-8e7d01022833-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.173576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.173544 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\") pod \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " Oct 30 22:30:20.173866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.173854 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\" (UniqueName: \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\") pod \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\" (UID: \"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3\") " Oct 30 22:30:20.173947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.173865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca" pod "9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" (UID: "9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.174539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.174518 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.176201 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.176177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv" pod "9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" (UID: "9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3"). InnerVolumeSpecName "kube-api-access-rtpbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.180810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.180781 5036 scope.go:122] "RemoveContainer" containerID="0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac" Oct 30 22:30:20.181184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.181161 5036 scope.go:122] "RemoveContainer" containerID="14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce" Oct 30 22:30:20.181356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.181335 5036 scope.go:122] "RemoveContainer" containerID="c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834" Oct 30 22:30:20.181446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.181422 5036 scope.go:122] "RemoveContainer" containerID="e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523" Oct 30 22:30:20.181810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.181786 5036 scope.go:122] "RemoveContainer" containerID="52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe" Oct 30 22:30:20.277744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.277669 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtpbv\" (UniqueName: \"kubernetes.io/projected/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3-kube-api-access-rtpbv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.709800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.709765 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:20.755552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.755523 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:20.779914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.773858 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerStarted","Data":"6e5fad04a29a6a402fca752a6abe6daa33980341220834c8ef95586655b8dbb9"} Oct 30 22:30:20.780916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.780482 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:20.784828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.784761 5036 generic.go:356] "Generic (PLEG): container finished" podID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerID="3620f1747130ed1759e2eba666c43d8b5f45f2f33ccf92f1cbe357bc2510b8f2" exitCode=60 Oct 30 22:30:20.784948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.784856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerDied","Data":"3620f1747130ed1759e2eba666c43d8b5f45f2f33ccf92f1cbe357bc2510b8f2"} Oct 30 22:30:20.784948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.784913 5036 scope.go:122] "RemoveContainer" containerID="42b1017c91c63942ec5a655be79c1229ca03acb34bb8a02ffa891510d84a0556" Oct 30 22:30:20.786305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.785461 5036 scope.go:122] "RemoveContainer" containerID="3620f1747130ed1759e2eba666c43d8b5f45f2f33ccf92f1cbe357bc2510b8f2" Oct 30 22:30:20.786305 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:20.785931 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc_odh-ai-gateway-infra(098d2d2e-595e-4d64-bd36-b9f72ad87d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" Oct 30 22:30:20.790065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.789799 5036 generic.go:356] "Generic (PLEG): container finished" podID="7e5a8505-b921-452e-8976-2921f116cbb2" containerID="71dadbd0795da56847cfb34cd0183f30b469bd1db9a690e8726f9857d959786e" exitCode=60 Oct 30 22:30:20.790065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.789882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerDied","Data":"71dadbd0795da56847cfb34cd0183f30b469bd1db9a690e8726f9857d959786e"} Oct 30 22:30:20.795417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.794964 5036 generic.go:356] "Generic (PLEG): container finished" podID="0530a85c-8a0f-417b-b430-92a298d192ab" containerID="e76aeae4b8fd6ce75e90329526485a97a60c8f700df877b07193e9722a61cf1a" exitCode=60 Oct 30 22:30:20.795417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.795026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerDied","Data":"e76aeae4b8fd6ce75e90329526485a97a60c8f700df877b07193e9722a61cf1a"} Oct 30 22:30:20.795592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.795450 5036 scope.go:122] "RemoveContainer" containerID="e76aeae4b8fd6ce75e90329526485a97a60c8f700df877b07193e9722a61cf1a" Oct 30 22:30:20.795968 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:20.795885 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-ddxws_odh-ai-gateway-infra(0530a85c-8a0f-417b-b430-92a298d192ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" Oct 30 22:30:20.799083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.798799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerStarted","Data":"85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29"} Oct 30 22:30:20.799249 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.799226 5036 scope.go:122] "RemoveContainer" containerID="85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29" Oct 30 22:30:20.799648 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:20.799627 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7_odh-ai-gateway-infra(a2fe75d3-cc21-4e33-96aa-5918f6c6260a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" Oct 30 22:30:20.801394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.801200 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:20.801940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.801918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" event={"ID":"718d3757-a41d-41ee-8d34-8e7d01022833","Type":"ContainerDied","Data":"465b69140f4dec0073c3a6ecb82c7f010fa6d05c9ab2132d485c7f269b6a0f6f"} Oct 30 22:30:20.802049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.802034 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8" Oct 30 22:30:20.805640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.805327 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" event={"ID":"9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3","Type":"ContainerDied","Data":"d1a9a0ca5c66337cc46d406924acd019581ece37547ba65b7c8322b509135dbb"} Oct 30 22:30:20.805640 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.805383 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx" Oct 30 22:30:20.808457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.807645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerStarted","Data":"80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632"} Oct 30 22:30:20.808457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.808058 5036 scope.go:122] "RemoveContainer" containerID="80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632" Oct 30 22:30:20.808457 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:20.808383 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96_odh-ai-gateway-infra(824b4b46-77b4-41a8-821f-8f0421ac3fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" Oct 30 22:30:20.828601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.823451 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerStarted","Data":"ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b"} Oct 30 22:30:20.828601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.824074 5036 scope.go:122] "RemoveContainer" containerID="ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b" Oct 30 22:30:20.828601 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:20.824615 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8_odh-ai-gateway-infra(bc84c7b4-6dc0-41d4-bd55-c807fd444c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" Oct 30 22:30:20.828601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.825911 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:20.831686 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.831660 5036 generic.go:356] "Generic (PLEG): container finished" podID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerID="b4b0a8a741cca81a4246f2d722b6a698f76ea4103c053227c4a800704f3e4d6a" exitCode=60 Oct 30 22:30:20.831821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.831805 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerDied","Data":"b4b0a8a741cca81a4246f2d722b6a698f76ea4103c053227c4a800704f3e4d6a"} Oct 30 22:30:20.957991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.946805 5036 scope.go:122] "RemoveContainer" containerID="cfa95e81a298426409c2288530a77c40c47d790b38b4b0b744582e7612249939" Oct 30 22:30:20.979879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:20.979849 5036 scope.go:122] "RemoveContainer" containerID="943b3a47c2ec8a88246c59f3b4e7076275716835370d4b94a650f0b683bd1844" Oct 30 22:30:21.023502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.022526 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:21.053511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.053481 5036 scope.go:122] "RemoveContainer" containerID="731977ff93e8f20f11c09ac52727d574804477fca1229a70ce8a9a6378a694ef" Oct 30 22:30:21.074395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.074364 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889990-8fbs8"] Oct 30 22:30:21.081414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.081389 5036 scope.go:122] "RemoveContainer" containerID="efaa9bf971608cf3b2516c184312080f2bbca13d103b698608afabfbdc89d0c9" Oct 30 22:30:21.084106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.083929 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:21.090247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.090223 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-18ba2f-29889990-qzhjx"] Oct 30 22:30:21.119847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.119825 5036 scope.go:122] "RemoveContainer" containerID="4db8242e4435254199162ab6010078c2f0c12323b3162649371a9bb0817a0301" Oct 30 22:30:21.122608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.122585 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:21.181089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.181056 5036 scope.go:122] "RemoveContainer" containerID="0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5" Oct 30 22:30:21.256498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.256464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\") pod \"e4b52730-29eb-49d4-8da6-9023f88646ad\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " Oct 30 22:30:21.256762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.256592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\" (UniqueName: \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\") pod \"e4b52730-29eb-49d4-8da6-9023f88646ad\" (UID: \"e4b52730-29eb-49d4-8da6-9023f88646ad\") " Oct 30 22:30:21.256936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.256909 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca" pod "e4b52730-29eb-49d4-8da6-9023f88646ad" (UID: "e4b52730-29eb-49d4-8da6-9023f88646ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.257730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.257710 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b52730-29eb-49d4-8da6-9023f88646ad-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.320870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.320827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m" pod "e4b52730-29eb-49d4-8da6-9023f88646ad" (UID: "e4b52730-29eb-49d4-8da6-9023f88646ad"). InnerVolumeSpecName "kube-api-access-kkr5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.360194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.360050 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkr5m\" (UniqueName: \"kubernetes.io/projected/e4b52730-29eb-49d4-8da6-9023f88646ad-kube-api-access-kkr5m\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.370901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.370872 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:21.374798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.374767 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.586856 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\" (UniqueName: \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\") pod \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.598813 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\") pod \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\" (UID: \"1dd0862d-5888-4022-9d6b-f4bf203c45ce\") " Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.599104 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\") pod \"7e5a8505-b921-452e-8976-2921f116cbb2\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.599209 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\" (UniqueName: \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\") pod \"7e5a8505-b921-452e-8976-2921f116cbb2\" (UID: \"7e5a8505-b921-452e-8976-2921f116cbb2\") " Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.605767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca" pod "1dd0862d-5888-4022-9d6b-f4bf203c45ce" (UID: "1dd0862d-5888-4022-9d6b-f4bf203c45ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.606748 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca" pod "7e5a8505-b921-452e-8976-2921f116cbb2" (UID: "7e5a8505-b921-452e-8976-2921f116cbb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.619317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.609886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8" pod "1dd0862d-5888-4022-9d6b-f4bf203c45ce" (UID: "1dd0862d-5888-4022-9d6b-f4bf203c45ce"). InnerVolumeSpecName "kube-api-access-j9rb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.642801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.642763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j" pod "7e5a8505-b921-452e-8976-2921f116cbb2" (UID: "7e5a8505-b921-452e-8976-2921f116cbb2"). InnerVolumeSpecName "kube-api-access-7kk5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.701906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.701879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a8505-b921-452e-8976-2921f116cbb2-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.702019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.701908 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kk5j\" (UniqueName: \"kubernetes.io/projected/7e5a8505-b921-452e-8976-2921f116cbb2-kube-api-access-7kk5j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.702019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.701920 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rb8\" (UniqueName: \"kubernetes.io/projected/1dd0862d-5888-4022-9d6b-f4bf203c45ce-kube-api-access-j9rb8\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.702019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.701929 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd0862d-5888-4022-9d6b-f4bf203c45ce-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.812661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.812628 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:21.850582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.850547 5036 generic.go:356] "Generic (PLEG): container finished" podID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerID="6e5fad04a29a6a402fca752a6abe6daa33980341220834c8ef95586655b8dbb9" exitCode=60 Oct 30 22:30:21.850780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.850636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerDied","Data":"6e5fad04a29a6a402fca752a6abe6daa33980341220834c8ef95586655b8dbb9"} Oct 30 22:30:21.850780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.850707 5036 scope.go:122] "RemoveContainer" containerID="0d61b0cba1160594f828443d88a4f721446e6b31758d282e96eaddcc70df85ca" Oct 30 22:30:21.857109 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.857079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" event={"ID":"7e5a8505-b921-452e-8976-2921f116cbb2","Type":"ContainerDied","Data":"234573849b93fcd2867cc9c0ffe4b34cc7007dc4ef5e05e06ddaf0919f613143"} Oct 30 22:30:21.857240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.857196 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf" Oct 30 22:30:21.867671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.867641 5036 generic.go:356] "Generic (PLEG): container finished" podID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerID="1f74cdb95e9f6fe815a7d3691dd7ef545371fb46aee5a030570d8fb53144927d" exitCode=60 Oct 30 22:30:21.868227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.867732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerDied","Data":"1f74cdb95e9f6fe815a7d3691dd7ef545371fb46aee5a030570d8fb53144927d"} Oct 30 22:30:21.868490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.868467 5036 scope.go:122] "RemoveContainer" containerID="1f74cdb95e9f6fe815a7d3691dd7ef545371fb46aee5a030570d8fb53144927d" Oct 30 22:30:21.868968 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.868941 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp_odh-ai-gateway-infra(5aa7c802-4abd-437b-abd0-2ab67b1f6f4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" Oct 30 22:30:21.873773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.873741 5036 generic.go:356] "Generic (PLEG): container finished" podID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerID="85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29" exitCode=60 Oct 30 22:30:21.873975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.873902 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerDied","Data":"85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29"} Oct 30 22:30:21.874048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.874014 5036 scope.go:122] "RemoveContainer" containerID="71dadbd0795da56847cfb34cd0183f30b469bd1db9a690e8726f9857d959786e" Oct 30 22:30:21.875522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.874623 5036 scope.go:122] "RemoveContainer" containerID="85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29" Oct 30 22:30:21.875624 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.875517 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7_odh-ai-gateway-infra(a2fe75d3-cc21-4e33-96aa-5918f6c6260a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" Oct 30 22:30:21.882455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.881151 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerID="a80cb653065aa907573ada8fd03f6b518c142d3c7f4448f2b1aaa79b3daf1717" exitCode=60 Oct 30 22:30:21.882455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.881232 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerDied","Data":"a80cb653065aa907573ada8fd03f6b518c142d3c7f4448f2b1aaa79b3daf1717"} Oct 30 22:30:21.882455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.881855 5036 scope.go:122] "RemoveContainer" containerID="a80cb653065aa907573ada8fd03f6b518c142d3c7f4448f2b1aaa79b3daf1717" Oct 30 22:30:21.882455 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.882292 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj_odh-ai-gateway-infra(1d3a9c92-5817-480c-9a5d-e024e6fec13f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" Oct 30 22:30:21.889190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.888514 5036 generic.go:356] "Generic (PLEG): container finished" podID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerID="80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632" exitCode=60 Oct 30 22:30:21.889190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.888597 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerDied","Data":"80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632"} Oct 30 22:30:21.889190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.889054 5036 scope.go:122] "RemoveContainer" containerID="80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632" Oct 30 22:30:21.889712 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.889672 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96_odh-ai-gateway-infra(824b4b46-77b4-41a8-821f-8f0421ac3fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" Oct 30 22:30:21.892790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.892050 5036 generic.go:356] "Generic (PLEG): container finished" podID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerID="ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b" exitCode=60 Oct 30 22:30:21.892790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.892116 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerDied","Data":"ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b"} Oct 30 22:30:21.893406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.893075 5036 scope.go:122] "RemoveContainer" containerID="e6b8c703ff4b034d4cad96aa92dce7ab40f3b4b8ded9bb7e0c9a8799391b7523" Oct 30 22:30:21.898029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.898009 5036 scope.go:122] "RemoveContainer" containerID="ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b" Oct 30 22:30:21.898602 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.898576 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8_odh-ai-gateway-infra(bc84c7b4-6dc0-41d4-bd55-c807fd444c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" Oct 30 22:30:21.916607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.916580 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:21.929712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.927611 5036 generic.go:356] "Generic (PLEG): container finished" podID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerID="b6a96d5f704a6ff50b7cfe40b788d71e01c619c71f7c4e044ddcc91d8a5c7560" exitCode=60 Oct 30 22:30:21.929712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.927725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" event={"ID":"d782b5c7-490f-4461-b0c9-8ac8a12e3462","Type":"ContainerDied","Data":"b6a96d5f704a6ff50b7cfe40b788d71e01c619c71f7c4e044ddcc91d8a5c7560"} Oct 30 22:30:21.929712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.928237 5036 scope.go:122] "RemoveContainer" containerID="b6a96d5f704a6ff50b7cfe40b788d71e01c619c71f7c4e044ddcc91d8a5c7560" Oct 30 22:30:21.929712 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.928717 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95_odh-ai-gateway-infra(d782b5c7-490f-4461-b0c9-8ac8a12e3462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" Oct 30 22:30:21.932533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.932508 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" Oct 30 22:30:21.934370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.933552 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5" event={"ID":"1dd0862d-5888-4022-9d6b-f4bf203c45ce","Type":"ContainerDied","Data":"f21c0338690a2aa1b6bc344a3106fd3ac2302ebbbaef7399708dcc7a0c94d718"} Oct 30 22:30:21.935357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.935335 5036 scope.go:122] "RemoveContainer" containerID="e7b865264b2a2dc6dd7307a988c6f54c5c5190749485f96fc7114936d03cfa5d" Oct 30 22:30:21.938315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.937885 5036 generic.go:356] "Generic (PLEG): container finished" podID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerID="4c9e6fcb21aa4203f8aa6beea32f833e363b266fb0a1840d37951af0ee441448" exitCode=60 Oct 30 22:30:21.938315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.938005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" event={"ID":"a5389037-db6e-4fa6-abfa-60c9e255cf77","Type":"ContainerDied","Data":"4c9e6fcb21aa4203f8aa6beea32f833e363b266fb0a1840d37951af0ee441448"} Oct 30 22:30:21.939581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.939076 5036 scope.go:122] "RemoveContainer" containerID="4c9e6fcb21aa4203f8aa6beea32f833e363b266fb0a1840d37951af0ee441448" Oct 30 22:30:21.939581 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.939514 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k_odh-ai-gateway-infra(a5389037-db6e-4fa6-abfa-60c9e255cf77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" Oct 30 22:30:21.948344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.948152 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" Oct 30 22:30:21.948743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.948686 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj" event={"ID":"e4b52730-29eb-49d4-8da6-9023f88646ad","Type":"ContainerDied","Data":"2cf174f66ad08151087f19874a4e85395a4459777a18d79886abb9dffbbd7b25"} Oct 30 22:30:21.957679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.954487 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:21.957679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.956507 5036 generic.go:356] "Generic (PLEG): container finished" podID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerID="a6c23944ed95c198af7f0459335abcaae19dde9c4e2a2953778e74194cd6aa90" exitCode=60 Oct 30 22:30:21.957679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.956558 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerDied","Data":"a6c23944ed95c198af7f0459335abcaae19dde9c4e2a2953778e74194cd6aa90"} Oct 30 22:30:21.957679 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.957067 5036 scope.go:122] "RemoveContainer" containerID="a6c23944ed95c198af7f0459335abcaae19dde9c4e2a2953778e74194cd6aa90" Oct 30 22:30:21.957679 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:21.957477 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t_odh-ai-gateway-infra(ed331f16-3f65-41f7-9907-50008df1ba4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" Oct 30 22:30:21.964588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.964530 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:21.978588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:21.978438 5036 scope.go:122] "RemoveContainer" containerID="c0ece8a775f62a1dc224e103d4c03af3f66a57660854ea83a21079ecdff06834" Oct 30 22:30:22.023377 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.021057 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:22.032670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.032607 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-38d6e2-29889990-j95xf"] Oct 30 22:30:22.037924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.037160 5036 scope.go:122] "RemoveContainer" containerID="2e72e3aecc4a8e2d84b2899b15e341565b80cdd4ab9591f316ebdde239675a62" Oct 30 22:30:22.045536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.045503 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:22.056062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.056042 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c92aec-29889990-hvhkj"] Oct 30 22:30:22.094950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.094922 5036 scope.go:122] "RemoveContainer" containerID="66b03422baca8e77073f282adea1dd6e65c83e7268f4fd4c3a880475992cf73e" Oct 30 22:30:22.126900 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.126869 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:22.138917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.138893 5036 scope.go:122] "RemoveContainer" containerID="52eb6280b52bd7712640769e32e0088d4d9f7d7218d29b3ec38a2bf0c91a39fe" Oct 30 22:30:22.175703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.175680 5036 scope.go:122] "RemoveContainer" containerID="b4b0a8a741cca81a4246f2d722b6a698f76ea4103c053227c4a800704f3e4d6a" Oct 30 22:30:22.181587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.181566 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:22.223319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.202391 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" path="/var/lib/kubelet/pods/718d3757-a41d-41ee-8d34-8e7d01022833/volumes" Oct 30 22:30:22.223319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.203152 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" path="/var/lib/kubelet/pods/7e5a8505-b921-452e-8976-2921f116cbb2/volumes" Oct 30 22:30:22.223319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.203940 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" path="/var/lib/kubelet/pods/9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3/volumes" Oct 30 22:30:22.223319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.205006 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" path="/var/lib/kubelet/pods/e4b52730-29eb-49d4-8da6-9023f88646ad/volumes" Oct 30 22:30:22.224145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.224111 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\") pod \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " Oct 30 22:30:22.224422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.224403 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\" (UniqueName: \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\") pod \"4b9090da-d338-4ba0-ada0-ea88934aacc5\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " Oct 30 22:30:22.224551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.224464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\") pod \"4b9090da-d338-4ba0-ada0-ea88934aacc5\" (UID: \"4b9090da-d338-4ba0-ada0-ea88934aacc5\") " Oct 30 22:30:22.224652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.224638 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\" (UniqueName: \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\") pod \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\" (UID: \"098d2d2e-595e-4d64-bd36-b9f72ad87d62\") " Oct 30 22:30:22.267166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.225970 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca" pod "098d2d2e-595e-4d64-bd36-b9f72ad87d62" (UID: "098d2d2e-595e-4d64-bd36-b9f72ad87d62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.267166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.254892 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca" pod "4b9090da-d338-4ba0-ada0-ea88934aacc5" (UID: "4b9090da-d338-4ba0-ada0-ea88934aacc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.303631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.294607 5036 scope.go:122] "RemoveContainer" containerID="0d7aac8e0d056d2164837ba04a4db7c3b58e7826f828d97f3c7d4eff748466ac" Oct 30 22:30:22.325423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.325365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj" pod "4b9090da-d338-4ba0-ada0-ea88934aacc5" (UID: "4b9090da-d338-4ba0-ada0-ea88934aacc5"). InnerVolumeSpecName "kube-api-access-vp6hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.330322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.330279 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098d2d2e-595e-4d64-bd36-b9f72ad87d62-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.330463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.330326 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp6hj\" (UniqueName: \"kubernetes.io/projected/4b9090da-d338-4ba0-ada0-ea88934aacc5-kube-api-access-vp6hj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.330543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.330345 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9090da-d338-4ba0-ada0-ea88934aacc5-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.332804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.332754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5" pod "098d2d2e-595e-4d64-bd36-b9f72ad87d62" (UID: "098d2d2e-595e-4d64-bd36-b9f72ad87d62"). InnerVolumeSpecName "kube-api-access-vmxw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.396142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.396105 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:22.410951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.410925 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:22.411319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.411210 5036 scope.go:122] "RemoveContainer" containerID="45d2ad9a2a370ada4c743aec3edcf386140bbf509dac6f400209dc91a15a16c5" Oct 30 22:30:22.417157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.417134 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26a128-29889990-pd7n5"] Oct 30 22:30:22.431231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.431196 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\" (UniqueName: \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\") pod \"0530a85c-8a0f-417b-b430-92a298d192ab\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " Oct 30 22:30:22.431403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.431365 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\") pod \"0530a85c-8a0f-417b-b430-92a298d192ab\" (UID: \"0530a85c-8a0f-417b-b430-92a298d192ab\") " Oct 30 22:30:22.432349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.432323 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmxw5\" (UniqueName: \"kubernetes.io/projected/098d2d2e-595e-4d64-bd36-b9f72ad87d62-kube-api-access-vmxw5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.432911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.432881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca" pod "0530a85c-8a0f-417b-b430-92a298d192ab" (UID: "0530a85c-8a0f-417b-b430-92a298d192ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.439492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.439468 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg" pod "0530a85c-8a0f-417b-b430-92a298d192ab" (UID: "0530a85c-8a0f-417b-b430-92a298d192ab"). InnerVolumeSpecName "kube-api-access-hgrtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.455241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.455217 5036 scope.go:122] "RemoveContainer" containerID="14cef2878ff1ff81624014175baa3ed15bd66f2669604ea944f300a11e4b54ce" Oct 30 22:30:22.534998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.534951 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrtg\" (UniqueName: \"kubernetes.io/projected/0530a85c-8a0f-417b-b430-92a298d192ab-kube-api-access-hgrtg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.534998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.534994 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0530a85c-8a0f-417b-b430-92a298d192ab-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.972355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.972245 5036 generic.go:356] "Generic (PLEG): container finished" podID="72557df9-139d-4974-8363-ca23eed5fcb4" containerID="8e654a31424b4d7b2adcdb6856ebe30334029727066bd895ad3682085bab22c4" exitCode=60 Oct 30 22:30:22.972355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.972308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerDied","Data":"8e654a31424b4d7b2adcdb6856ebe30334029727066bd895ad3682085bab22c4"} Oct 30 22:30:22.972593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.972410 5036 scope.go:122] "RemoveContainer" containerID="0b17ac80c2d78ae6ed123fd2094603dfb29f376e36089fd71f5f1f5acedd0af5" Oct 30 22:30:22.972974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.972953 5036 scope.go:122] "RemoveContainer" containerID="8e654a31424b4d7b2adcdb6856ebe30334029727066bd895ad3682085bab22c4" Oct 30 22:30:22.973541 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:22.973453 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25_odh-ai-gateway-infra(72557df9-139d-4974-8363-ca23eed5fcb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" Oct 30 22:30:22.974132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.974104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" event={"ID":"4b9090da-d338-4ba0-ada0-ea88934aacc5","Type":"ContainerDied","Data":"b0b9e2eaef02eefdf7f716d4075060cea8847c5eb9aff4100badd2c385a61110"} Oct 30 22:30:22.974248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.974139 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j" Oct 30 22:30:22.975864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.975833 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" Oct 30 22:30:22.976387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.976368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc" event={"ID":"098d2d2e-595e-4d64-bd36-b9f72ad87d62","Type":"ContainerDied","Data":"217c13d6c63e99d162737545b0908ecf6e47ff1e4c065e5bd3c8b8b2217ae939"} Oct 30 22:30:22.980825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.980806 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" Oct 30 22:30:22.980940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.980835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws" event={"ID":"0530a85c-8a0f-417b-b430-92a298d192ab","Type":"ContainerDied","Data":"c08baf81744c212fdc3849cd0b4d6b13dc18e28b6fb51dd17af38212e42fa23a"} Oct 30 22:30:22.998233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:22.997117 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:23.003245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.003217 5036 scope.go:122] "RemoveContainer" containerID="6e5fad04a29a6a402fca752a6abe6daa33980341220834c8ef95586655b8dbb9" Oct 30 22:30:23.132190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.132162 5036 scope.go:122] "RemoveContainer" containerID="3620f1747130ed1759e2eba666c43d8b5f45f2f33ccf92f1cbe357bc2510b8f2" Oct 30 22:30:23.142597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.142007 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:23.176800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.172234 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:23.177393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.177369 5036 scope.go:122] "RemoveContainer" containerID="e76aeae4b8fd6ce75e90329526485a97a60c8f700df877b07193e9722a61cf1a" Oct 30 22:30:23.230846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.228872 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:23.230846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.228923 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28d39b-29889990-kzf6j"] Oct 30 22:30:23.247099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.245610 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:23.255291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.254364 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:23.281961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.279747 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:23.298388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.298356 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecfbd6-29889990-cdffc"] Oct 30 22:30:23.315615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.306120 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:23.322166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.322133 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-ddxws"] Oct 30 22:30:23.437907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.436121 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:23.561175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.561144 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\") pod \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " Oct 30 22:30:23.561946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.561912 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca" pod "a2fe75d3-cc21-4e33-96aa-5918f6c6260a" (UID: "a2fe75d3-cc21-4e33-96aa-5918f6c6260a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.562084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.561969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\" (UniqueName: \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\") pod \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\" (UID: \"a2fe75d3-cc21-4e33-96aa-5918f6c6260a\") " Oct 30 22:30:23.563461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.563432 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.566372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.566338 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg" pod "a2fe75d3-cc21-4e33-96aa-5918f6c6260a" (UID: "a2fe75d3-cc21-4e33-96aa-5918f6c6260a"). InnerVolumeSpecName "kube-api-access-bcptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.682355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.676045 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcptg\" (UniqueName: \"kubernetes.io/projected/a2fe75d3-cc21-4e33-96aa-5918f6c6260a-kube-api-access-bcptg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.695367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.695340 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:23.704543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.704517 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:23.818971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.818832 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:23.839951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.839862 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:23.845534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.845507 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:23.873446 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.873422 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.889684 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\") pod \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.889818 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\") pod \"824b4b46-77b4-41a8-821f-8f0421ac3fab\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.890221 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca" pod "5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" (UID: "5aa7c802-4abd-437b-abd0-2ab67b1f6f4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.890331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\" (UniqueName: \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\") pod \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\" (UID: \"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d\") " Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.890401 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\" (UniqueName: \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\") pod \"824b4b46-77b4-41a8-821f-8f0421ac3fab\" (UID: \"824b4b46-77b4-41a8-821f-8f0421ac3fab\") " Oct 30 22:30:23.890645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.890602 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca" pod "824b4b46-77b4-41a8-821f-8f0421ac3fab" (UID: "824b4b46-77b4-41a8-821f-8f0421ac3fab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.893217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.892403 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.893217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.892442 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/824b4b46-77b4-41a8-821f-8f0421ac3fab-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.921505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.921460 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh" pod "5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" (UID: "5aa7c802-4abd-437b-abd0-2ab67b1f6f4d"). InnerVolumeSpecName "kube-api-access-jzvhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.924469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.924426 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:23.936127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.935587 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs" pod "824b4b46-77b4-41a8-821f-8f0421ac3fab" (UID: "824b4b46-77b4-41a8-821f-8f0421ac3fab"). InnerVolumeSpecName "kube-api-access-xlbbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.993696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.993661 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\") pod \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " Oct 30 22:30:23.993877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.993771 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\") pod \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " Oct 30 22:30:23.993963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.993880 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\" (UniqueName: \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\") pod \"ed331f16-3f65-41f7-9907-50008df1ba4b\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " Oct 30 22:30:23.993963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.993932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\") pod \"a5389037-db6e-4fa6-abfa-60c9e255cf77\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " Oct 30 22:30:23.994082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\" (UniqueName: \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\") pod \"a5389037-db6e-4fa6-abfa-60c9e255cf77\" (UID: \"a5389037-db6e-4fa6-abfa-60c9e255cf77\") " Oct 30 22:30:23.994136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\") pod \"ed331f16-3f65-41f7-9907-50008df1ba4b\" (UID: \"ed331f16-3f65-41f7-9907-50008df1ba4b\") " Oct 30 22:30:23.994136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994018 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca" pod "bc84c7b4-6dc0-41d4-bd55-c807fd444c69" (UID: "bc84c7b4-6dc0-41d4-bd55-c807fd444c69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.994279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\" (UniqueName: \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\") pod \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\" (UID: \"bc84c7b4-6dc0-41d4-bd55-c807fd444c69\") " Oct 30 22:30:23.994362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994311 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\" (UniqueName: \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\") pod \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\" (UID: \"d782b5c7-490f-4461-b0c9-8ac8a12e3462\") " Oct 30 22:30:23.994472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994447 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca" pod "a5389037-db6e-4fa6-abfa-60c9e255cf77" (UID: "a5389037-db6e-4fa6-abfa-60c9e255cf77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.994557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994511 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca" pod "ed331f16-3f65-41f7-9907-50008df1ba4b" (UID: "ed331f16-3f65-41f7-9907-50008df1ba4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.994661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.994631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca" pod "d782b5c7-490f-4461-b0c9-8ac8a12e3462" (UID: "d782b5c7-490f-4461-b0c9-8ac8a12e3462"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:23.995420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995392 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.995552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995423 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvhh\" (UniqueName: \"kubernetes.io/projected/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d-kube-api-access-jzvhh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.995552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995440 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d782b5c7-490f-4461-b0c9-8ac8a12e3462-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.995552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995456 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbbs\" (UniqueName: \"kubernetes.io/projected/824b4b46-77b4-41a8-821f-8f0421ac3fab-kube-api-access-xlbbs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.995552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995470 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5389037-db6e-4fa6-abfa-60c9e255cf77-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.995552 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.995482 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed331f16-3f65-41f7-9907-50008df1ba4b-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.996970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.996919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch" pod "bc84c7b4-6dc0-41d4-bd55-c807fd444c69" (UID: "bc84c7b4-6dc0-41d4-bd55-c807fd444c69"). InnerVolumeSpecName "kube-api-access-w7rch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.996970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.996953 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc" pod "ed331f16-3f65-41f7-9907-50008df1ba4b" (UID: "ed331f16-3f65-41f7-9907-50008df1ba4b"). InnerVolumeSpecName "kube-api-access-t2wgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.997551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.997513 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4" pod "a5389037-db6e-4fa6-abfa-60c9e255cf77" (UID: "a5389037-db6e-4fa6-abfa-60c9e255cf77"). InnerVolumeSpecName "kube-api-access-rclz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:23.997644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:23.997566 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z" pod "d782b5c7-490f-4461-b0c9-8ac8a12e3462" (UID: "d782b5c7-490f-4461-b0c9-8ac8a12e3462"). InnerVolumeSpecName "kube-api-access-qdd8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:24.003473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.003448 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:24.046404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.046369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" event={"ID":"5aa7c802-4abd-437b-abd0-2ab67b1f6f4d","Type":"ContainerDied","Data":"ea4182654575d83767bc38522969ef107ceef6d89a50602b570bb8a35eebd103"} Oct 30 22:30:24.046591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.046382 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp" Oct 30 22:30:24.046591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.046430 5036 scope.go:122] "RemoveContainer" containerID="1f74cdb95e9f6fe815a7d3691dd7ef545371fb46aee5a030570d8fb53144927d" Oct 30 22:30:24.048551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.048514 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" Oct 30 22:30:24.048551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.048541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7" event={"ID":"a2fe75d3-cc21-4e33-96aa-5918f6c6260a","Type":"ContainerDied","Data":"b1d3d902ef049e3139f7516830408808244adf9811092f8aea507eba38b887e1"} Oct 30 22:30:24.051117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.051098 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" Oct 30 22:30:24.051561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.051538 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj" event={"ID":"1d3a9c92-5817-480c-9a5d-e024e6fec13f","Type":"ContainerDied","Data":"954b7ff56a22c1ec9daba6a5d9853b18eef7b161bbb7f6ea26f276d5dd6fdea5"} Oct 30 22:30:24.055882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.055854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" event={"ID":"824b4b46-77b4-41a8-821f-8f0421ac3fab","Type":"ContainerDied","Data":"3db168262723dd1d9643eec1144ee0c250253dc3e64b0d3d1d7ef9f515bdf360"} Oct 30 22:30:24.056025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.055958 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96" Oct 30 22:30:24.063166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.063146 5036 scope.go:122] "RemoveContainer" containerID="85e23afc6d50910e65bdafaef09c897a4a80d1e8b07c5245b14dd2a4709ebd29" Oct 30 22:30:24.067403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.067188 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" Oct 30 22:30:24.067403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.067196 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8" event={"ID":"bc84c7b4-6dc0-41d4-bd55-c807fd444c69","Type":"ContainerDied","Data":"7cebdd3679196c7b69d95181ae855a9563fc37bc976ca9c1393de572c31d736c"} Oct 30 22:30:24.069331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.069229 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" event={"ID":"d782b5c7-490f-4461-b0c9-8ac8a12e3462","Type":"ContainerDied","Data":"1fbfbf85b0aed0c01c10f49b876b68fee2588247e060db3584e6977ac85a9128"} Oct 30 22:30:24.069430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.069381 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95" Oct 30 22:30:24.089501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.089476 5036 scope.go:122] "RemoveContainer" containerID="a80cb653065aa907573ada8fd03f6b518c142d3c7f4448f2b1aaa79b3daf1717" Oct 30 22:30:24.091183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.091157 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" Oct 30 22:30:24.091321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.091155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k" event={"ID":"a5389037-db6e-4fa6-abfa-60c9e255cf77","Type":"ContainerDied","Data":"3485a6e2bbac4b0fcdc5dbd09d4b5c89841c5ce1687ae7ec776492a26decce4f"} Oct 30 22:30:24.092958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.092939 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" Oct 30 22:30:24.093248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.092959 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t" event={"ID":"ed331f16-3f65-41f7-9907-50008df1ba4b","Type":"ContainerDied","Data":"b6a5a4befd1bd93e4f2a2a895a4a1a0ca2e084db9c0420ee05147b666249dd5e"} Oct 30 22:30:24.097065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.097040 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\" (UniqueName: \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\") pod \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " Oct 30 22:30:24.097258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.097234 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\") pod \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\" (UID: \"1d3a9c92-5817-480c-9a5d-e024e6fec13f\") " Oct 30 22:30:24.098117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.098091 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7rch\" (UniqueName: \"kubernetes.io/projected/bc84c7b4-6dc0-41d4-bd55-c807fd444c69-kube-api-access-w7rch\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.098213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.098122 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdd8z\" (UniqueName: \"kubernetes.io/projected/d782b5c7-490f-4461-b0c9-8ac8a12e3462-kube-api-access-qdd8z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.098213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.098139 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wgc\" (UniqueName: \"kubernetes.io/projected/ed331f16-3f65-41f7-9907-50008df1ba4b-kube-api-access-t2wgc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.098213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.098155 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclz4\" (UniqueName: \"kubernetes.io/projected/a5389037-db6e-4fa6-abfa-60c9e255cf77-kube-api-access-rclz4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.098406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.098251 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca" pod "1d3a9c92-5817-480c-9a5d-e024e6fec13f" (UID: "1d3a9c92-5817-480c-9a5d-e024e6fec13f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:24.101114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.101082 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb" pod "1d3a9c92-5817-480c-9a5d-e024e6fec13f" (UID: "1d3a9c92-5817-480c-9a5d-e024e6fec13f"). InnerVolumeSpecName "kube-api-access-9bdqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:24.106015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.105983 5036 scope.go:122] "RemoveContainer" containerID="80e6ef34e0a7721268b82c7e127da9021c97d9f874430257d71c2740118d5632" Oct 30 22:30:24.118737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.118719 5036 scope.go:122] "RemoveContainer" containerID="ddbb975849cdbd95ed5aa953ab8d8fed838aa881ff02c5071dd22e5bb7206c7b" Oct 30 22:30:24.129011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.128982 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:24.133952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.133848 5036 scope.go:122] "RemoveContainer" containerID="b6a96d5f704a6ff50b7cfe40b788d71e01c619c71f7c4e044ddcc91d8a5c7560" Oct 30 22:30:24.135892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.135868 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f60026-29889990-cph96"] Oct 30 22:30:24.150136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.150117 5036 scope.go:122] "RemoveContainer" containerID="4c9e6fcb21aa4203f8aa6beea32f833e363b266fb0a1840d37951af0ee441448" Oct 30 22:30:24.152194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.152165 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:24.162815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.162792 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cca1c7-29889990-2d7f7"] Oct 30 22:30:24.164482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.164460 5036 scope.go:122] "RemoveContainer" containerID="a6c23944ed95c198af7f0459335abcaae19dde9c4e2a2953778e74194cd6aa90" Oct 30 22:30:24.202026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.201998 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3a9c92-5817-480c-9a5d-e024e6fec13f-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.202626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.202581 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdqb\" (UniqueName: \"kubernetes.io/projected/1d3a9c92-5817-480c-9a5d-e024e6fec13f-kube-api-access-9bdqb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.225416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.225382 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" path="/var/lib/kubelet/pods/0530a85c-8a0f-417b-b430-92a298d192ab/volumes" Oct 30 22:30:24.226493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.226472 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" path="/var/lib/kubelet/pods/098d2d2e-595e-4d64-bd36-b9f72ad87d62/volumes" Oct 30 22:30:24.228340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.228320 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" path="/var/lib/kubelet/pods/1dd0862d-5888-4022-9d6b-f4bf203c45ce/volumes" Oct 30 22:30:24.229652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.229633 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" path="/var/lib/kubelet/pods/4b9090da-d338-4ba0-ada0-ea88934aacc5/volumes" Oct 30 22:30:24.230399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.230381 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" path="/var/lib/kubelet/pods/824b4b46-77b4-41a8-821f-8f0421ac3fab/volumes" Oct 30 22:30:24.236786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.236766 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" path="/var/lib/kubelet/pods/a2fe75d3-cc21-4e33-96aa-5918f6c6260a/volumes" Oct 30 22:30:24.239253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.239234 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:24.239429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.239417 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5bdf-29889990-ql4s8"] Oct 30 22:30:24.239529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.239520 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:24.248030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.247980 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889990-t69kp"] Oct 30 22:30:24.271838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.271788 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:24.277605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.277551 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4b2bb0-29889990-4pl7t"] Oct 30 22:30:24.282821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.282796 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:24.283290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.283256 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:24.287779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.287758 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0a2577-29889990-kwk2k"] Oct 30 22:30:24.299356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.299324 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:24.303659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.303638 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df200a-29889990-n8r95"] Oct 30 22:30:24.385645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.385110 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:24.398765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.394679 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d5fa-29889990-4rdxj"] Oct 30 22:30:24.405571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.405548 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\") pod \"72557df9-139d-4974-8363-ca23eed5fcb4\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " Oct 30 22:30:24.405963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.405939 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca" pod "72557df9-139d-4974-8363-ca23eed5fcb4" (UID: "72557df9-139d-4974-8363-ca23eed5fcb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:24.406136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.405941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\" (UniqueName: \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\") pod \"72557df9-139d-4974-8363-ca23eed5fcb4\" (UID: \"72557df9-139d-4974-8363-ca23eed5fcb4\") " Oct 30 22:30:24.406678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.406658 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72557df9-139d-4974-8363-ca23eed5fcb4-maas-api-service-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:24.408749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.408722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w" pod "72557df9-139d-4974-8363-ca23eed5fcb4" (UID: "72557df9-139d-4974-8363-ca23eed5fcb4"). InnerVolumeSpecName "kube-api-access-l6p8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:24.519292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:24.517721 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6p8w\" (UniqueName: \"kubernetes.io/projected/72557df9-139d-4974-8363-ca23eed5fcb4-kube-api-access-l6p8w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:25.119954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:25.119920 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" Oct 30 22:30:25.119954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:25.119941 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25" event={"ID":"72557df9-139d-4974-8363-ca23eed5fcb4","Type":"ContainerDied","Data":"06802e8c49fe57fa3355d87e258515925e77b7b4390486fcdb3e12f081695bf4"} Oct 30 22:30:25.120226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:25.120006 5036 scope.go:122] "RemoveContainer" containerID="8e654a31424b4d7b2adcdb6856ebe30334029727066bd895ad3682085bab22c4" Oct 30 22:30:25.215157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:25.215113 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:25.220773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:25.220744 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889990-x8d25"] Oct 30 22:30:26.194608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.194577 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" path="/var/lib/kubelet/pods/1d3a9c92-5817-480c-9a5d-e024e6fec13f/volumes" Oct 30 22:30:26.195229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.195206 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" path="/var/lib/kubelet/pods/5aa7c802-4abd-437b-abd0-2ab67b1f6f4d/volumes" Oct 30 22:30:26.195729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.195707 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" path="/var/lib/kubelet/pods/72557df9-139d-4974-8363-ca23eed5fcb4/volumes" Oct 30 22:30:26.196594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.196573 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" path="/var/lib/kubelet/pods/a5389037-db6e-4fa6-abfa-60c9e255cf77/volumes" Oct 30 22:30:26.197051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.197031 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" path="/var/lib/kubelet/pods/bc84c7b4-6dc0-41d4-bd55-c807fd444c69/volumes" Oct 30 22:30:26.197541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.197518 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" path="/var/lib/kubelet/pods/d782b5c7-490f-4461-b0c9-8ac8a12e3462/volumes" Oct 30 22:30:26.198443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:26.198425 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" path="/var/lib/kubelet/pods/ed331f16-3f65-41f7-9907-50008df1ba4b/volumes" Oct 30 22:30:31.593744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.593709 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw"] Oct 30 22:30:31.594588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594570 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.594588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594586 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594603 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594611 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594620 5036 cpu_manager.go:414] "containerMap: removing container" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594629 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594641 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594648 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594661 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594666 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594673 5036 cpu_manager.go:414] "containerMap: removing container" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594677 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594685 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594690 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594696 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594701 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594708 5036 cpu_manager.go:414] "containerMap: removing container" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594712 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594723 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594728 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594733 5036 cpu_manager.go:414] "containerMap: removing container" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594737 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594744 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594750 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594759 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594764 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594773 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594778 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.594768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594787 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594791 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594805 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594810 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594817 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594821 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594831 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594835 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594841 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594846 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594852 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594857 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594862 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594866 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594876 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594880 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594887 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594891 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594896 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594901 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594907 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594912 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594917 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594921 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594926 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594930 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594945 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594949 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594957 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594961 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594970 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594974 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594981 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594985 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.594996 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595001 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595007 5036 cpu_manager.go:414] "containerMap: removing container" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595011 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595018 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595023 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595030 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595034 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595041 5036 cpu_manager.go:414] "containerMap: removing container" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595046 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595059 5036 cpu_manager.go:414] "containerMap: removing container" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595063 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595071 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595075 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595082 5036 cpu_manager.go:414] "containerMap: removing container" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595086 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595094 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595107 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595114 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595118 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595126 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595131 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595136 5036 cpu_manager.go:414] "containerMap: removing container" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595140 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595309 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595319 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595326 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595334 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595340 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595346 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595356 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595363 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595368 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595374 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595379 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595384 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595390 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595398 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595404 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595409 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595416 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed331f16-3f65-41f7-9907-50008df1ba4b" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595423 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc84c7b4-6dc0-41d4-bd55-c807fd444c69" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595431 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595438 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595444 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595449 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595455 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595463 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595469 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="824b4b46-77b4-41a8-821f-8f0421ac3fab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595476 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595482 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595489 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595495 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595501 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595506 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d782b5c7-490f-4461-b0c9-8ac8a12e3462" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595514 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595519 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595526 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595531 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595538 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595545 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595551 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595558 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595562 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595569 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595574 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:31.596416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.595580 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:31.602073 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.601515 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.606531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.606511 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:31.607346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.606830 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:31.607456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.607138 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:30:31.607456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.607161 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:31.608723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.608705 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw"] Oct 30 22:30:31.679068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.679032 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.679284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.679230 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6fn\" (UniqueName: \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.679356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.679328 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.780681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.780652 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.780856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.780781 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6fn\" (UniqueName: \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.780856 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.780820 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.781302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.781282 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.783467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.783445 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.792953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.792923 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6fn\" (UniqueName: \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\") pod \"maas-api-revoke-sub-df430947-567cw\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:31.922785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:31.922745 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:32.241152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.241125 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw"] Oct 30 22:30:32.242241 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:32.242215 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702d1680_ebbd_4795_ad9e_5b88cfb9a1d9.slice/crio-5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8 WatchSource:0}: Error finding container 5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8: Status 404 returned error can't find the container with id 5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8 Oct 30 22:30:32.877717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.877644 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv"] Oct 30 22:30:32.878518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878504 5036 cpu_manager.go:414] "containerMap: removing container" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878520 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718d3757-a41d-41ee-8d34-8e7d01022833" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878531 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878536 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878548 5036 cpu_manager.go:414] "containerMap: removing container" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878555 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098d2d2e-595e-4d64-bd36-b9f72ad87d62" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878565 5036 cpu_manager.go:414] "containerMap: removing container" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:32.878579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878569 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72557df9-139d-4974-8363-ca23eed5fcb4" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878589 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878594 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878607 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878612 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a8505-b921-452e-8976-2921f116cbb2" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878619 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878625 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3a9c92-5817-480c-9a5d-e024e6fec13f" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878657 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878666 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878808 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5389037-db6e-4fa6-abfa-60c9e255cf77" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878816 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f32f5cd-e8dd-4b75-b0df-c9c6fac1cfe3" containerName="cleanup" Oct 30 22:30:32.878824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878827 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b52730-29eb-49d4-8da6-9023f88646ad" containerName="cleanup" Oct 30 22:30:32.879159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878837 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0530a85c-8a0f-417b-b430-92a298d192ab" containerName="cleanup" Oct 30 22:30:32.879159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878843 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fe75d3-cc21-4e33-96aa-5918f6c6260a" containerName="cleanup" Oct 30 22:30:32.879159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878849 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7c802-4abd-437b-abd0-2ab67b1f6f4d" containerName="cleanup" Oct 30 22:30:32.879159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878856 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9090da-d338-4ba0-ada0-ea88934aacc5" containerName="cleanup" Oct 30 22:30:32.879159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.878864 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd0862d-5888-4022-9d6b-f4bf203c45ce" containerName="cleanup" Oct 30 22:30:32.882965 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.882947 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:32.891336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.891309 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv"] Oct 30 22:30:32.996297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.992176 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:32.996297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.992360 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:32.996297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:32.992427 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.094162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.094126 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.094386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.094234 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.094386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.094293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.094981 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.094957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.096864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.096834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.106664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.106640 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\") pod \"maas-api-revoke-sub-e5c3f31a-wrzpv\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.201821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.201792 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:33.264065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.264017 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" event={"ID":"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9","Type":"ContainerStarted","Data":"2e8468acccdeff011c579ae5afba49fc13950b799629195538afd7e660a3bbaa"} Oct 30 22:30:33.266807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.266769 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" event={"ID":"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9","Type":"ContainerStarted","Data":"5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8"} Oct 30 22:30:33.291726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.291663 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" podStartSLOduration=2.291641456 podStartE2EDuration="2.291641456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:33.285021916 +0000 UTC m=+3057.234976864" watchObservedRunningTime="2026-10-30 22:30:33.291641456 +0000 UTC m=+3057.241596406" Oct 30 22:30:33.516824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:33.516789 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv"] Oct 30 22:30:33.517744 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:33.517710 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3add391c_b5c1_479e_8390_b79be8a98a5c.slice/crio-f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa WatchSource:0}: Error finding container f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa: Status 404 returned error can't find the container with id f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa Oct 30 22:30:34.275254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:34.275211 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" event={"ID":"3add391c-b5c1-479e-8390-b79be8a98a5c","Type":"ContainerStarted","Data":"87a544e386793e69fb7715a35f5e6004c21eb9d30ddcfc82835f2cbe7ef78b1b"} Oct 30 22:30:34.275703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:34.275300 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" event={"ID":"3add391c-b5c1-479e-8390-b79be8a98a5c","Type":"ContainerStarted","Data":"f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa"} Oct 30 22:30:34.277170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:34.277145 5036 generic.go:356] "Generic (PLEG): container finished" podID="702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" containerID="2e8468acccdeff011c579ae5afba49fc13950b799629195538afd7e660a3bbaa" exitCode=0 Oct 30 22:30:34.277337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:34.277238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" event={"ID":"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9","Type":"ContainerDied","Data":"2e8468acccdeff011c579ae5afba49fc13950b799629195538afd7e660a3bbaa"} Oct 30 22:30:34.298077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:34.298035 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" podStartSLOduration=2.298022202 podStartE2EDuration="2.298022202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:34.292848757 +0000 UTC m=+3058.242803703" watchObservedRunningTime="2026-10-30 22:30:34.298022202 +0000 UTC m=+3058.247977147" Oct 30 22:30:35.290705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.290671 5036 generic.go:356] "Generic (PLEG): container finished" podID="3add391c-b5c1-479e-8390-b79be8a98a5c" containerID="87a544e386793e69fb7715a35f5e6004c21eb9d30ddcfc82835f2cbe7ef78b1b" exitCode=0 Oct 30 22:30:35.291110 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.290758 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" event={"ID":"3add391c-b5c1-479e-8390-b79be8a98a5c","Type":"ContainerDied","Data":"87a544e386793e69fb7715a35f5e6004c21eb9d30ddcfc82835f2cbe7ef78b1b"} Oct 30 22:30:35.500225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.500201 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:35.659166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.659128 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\") pod \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " Oct 30 22:30:35.659396 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.659341 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\" (UniqueName: \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\") pod \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " Oct 30 22:30:35.659649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.659620 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\") pod \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\" (UID: \"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9\") " Oct 30 22:30:35.660049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.660025 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca" pod "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" (UID: "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:35.660411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.660395 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:35.662052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.662020 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca" pod "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" (UID: "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:35.662141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.662082 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn" pod "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" (UID: "702d1680-ebbd-4795-ad9e-5b88cfb9a1d9"). InnerVolumeSpecName "kube-api-access-xt6fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:35.762473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.762441 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:35.762473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:35.762471 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt6fn\" (UniqueName: \"kubernetes.io/projected/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9-kube-api-access-xt6fn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:36.302515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.302485 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" Oct 30 22:30:36.303258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.302535 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw" event={"ID":"702d1680-ebbd-4795-ad9e-5b88cfb9a1d9","Type":"ContainerDied","Data":"5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8"} Oct 30 22:30:36.303258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.302601 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba86e02972f19fd890badf9dd75f0f544dce7c4b3971bcc8f982f64cdf2bcc8" Oct 30 22:30:36.532733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.532703 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:36.679421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.679307 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\") pod \"3add391c-b5c1-479e-8390-b79be8a98a5c\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " Oct 30 22:30:36.679648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.679430 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\") pod \"3add391c-b5c1-479e-8390-b79be8a98a5c\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " Oct 30 22:30:36.679730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.679709 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\") pod \"3add391c-b5c1-479e-8390-b79be8a98a5c\" (UID: \"3add391c-b5c1-479e-8390-b79be8a98a5c\") " Oct 30 22:30:36.680623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.680543 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca" pod "3add391c-b5c1-479e-8390-b79be8a98a5c" (UID: "3add391c-b5c1-479e-8390-b79be8a98a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:36.680763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.680749 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:36.682793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.682761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w" pod "3add391c-b5c1-479e-8390-b79be8a98a5c" (UID: "3add391c-b5c1-479e-8390-b79be8a98a5c"). InnerVolumeSpecName "kube-api-access-h8r4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:36.683019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.682994 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca" pod "3add391c-b5c1-479e-8390-b79be8a98a5c" (UID: "3add391c-b5c1-479e-8390-b79be8a98a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:36.784343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.784059 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r4w\" (UniqueName: \"kubernetes.io/projected/3add391c-b5c1-479e-8390-b79be8a98a5c-kube-api-access-h8r4w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:36.784343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:36.784102 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add391c-b5c1-479e-8390-b79be8a98a5c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:37.315040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:37.315005 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" Oct 30 22:30:37.315563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:37.315007 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv" event={"ID":"3add391c-b5c1-479e-8390-b79be8a98a5c","Type":"ContainerDied","Data":"f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa"} Oct 30 22:30:37.315563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:37.315171 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9bd430022701fec4165c41244dd36b1d2c89d3a976660bd0a1b7907b1b552aa" Oct 30 22:30:43.089820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.089784 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb"] Oct 30 22:30:43.090675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090657 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3add391c-b5c1-479e-8390-b79be8a98a5c" containerName="revoke-keys" Oct 30 22:30:43.090675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090675 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3add391c-b5c1-479e-8390-b79be8a98a5c" containerName="revoke-keys" Oct 30 22:30:43.090771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090716 5036 cpu_manager.go:414] "containerMap: removing container" podUID="702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" containerName="revoke-keys" Oct 30 22:30:43.090771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090721 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" containerName="revoke-keys" Oct 30 22:30:43.090890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090881 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3add391c-b5c1-479e-8390-b79be8a98a5c" containerName="revoke-keys" Oct 30 22:30:43.090950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.090899 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" containerName="revoke-keys" Oct 30 22:30:43.095219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.095200 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.100601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.100581 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:43.100904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.100866 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:30:43.108258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.108228 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb"] Oct 30 22:30:43.114373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.114345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxrb\" (UniqueName: \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.114973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.114610 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.114973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.114798 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.126341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.121571 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:43.126341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.121655 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:43.230290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.228508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.230290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.228698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxrb\" (UniqueName: \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.230290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.229054 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.239393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.232835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.251104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.251071 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.265634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.265231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxrb\" (UniqueName: \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\") pod \"maas-api-revoke-sub-99582201-ffhwb\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.413799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.413766 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:43.712347 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:43.712311 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf65d1cf_0e92_4ad7_8cc2_999c2e47b26d.slice/crio-8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780 WatchSource:0}: Error finding container 8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780: Status 404 returned error can't find the container with id 8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780 Oct 30 22:30:43.712508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:43.712396 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb"] Oct 30 22:30:44.464289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.459072 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" event={"ID":"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d","Type":"ContainerStarted","Data":"0bf2f4ff20e54128c123493ed3dbdce5f789fffe3db864db3aaa15f778c5b007"} Oct 30 22:30:44.464289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.459145 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" event={"ID":"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d","Type":"ContainerStarted","Data":"8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780"} Oct 30 22:30:44.513300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.508696 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" podStartSLOduration=1.508676886 podStartE2EDuration="1.508676886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:44.505577535 +0000 UTC m=+3068.455532532" watchObservedRunningTime="2026-10-30 22:30:44.508676886 +0000 UTC m=+3068.458631835" Oct 30 22:30:44.685712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.685676 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b"] Oct 30 22:30:44.690761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.690739 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.701483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.701454 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b"] Oct 30 22:30:44.761379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.761301 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.761626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.761608 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.761749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.761729 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwrk\" (UniqueName: \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.876296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.864977 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.891290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.876636 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwrk\" (UniqueName: \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.891721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.891689 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.892948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.892919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.897288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.893943 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:44.914320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:44.912964 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwrk\" (UniqueName: \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\") pod \"maas-api-revoke-sub-2318e2af-mz72b\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:45.030345 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:45.030234 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:45.326675 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:45.326642 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9cec06_7a60_4a6d_a2ea_eea4063436c6.slice/crio-9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05 WatchSource:0}: Error finding container 9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05: Status 404 returned error can't find the container with id 9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05 Oct 30 22:30:45.327363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:45.327323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b"] Oct 30 22:30:45.472197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:45.472165 5036 generic.go:356] "Generic (PLEG): container finished" podID="af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" containerID="0bf2f4ff20e54128c123493ed3dbdce5f789fffe3db864db3aaa15f778c5b007" exitCode=0 Oct 30 22:30:45.472755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:45.472309 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" event={"ID":"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d","Type":"ContainerDied","Data":"0bf2f4ff20e54128c123493ed3dbdce5f789fffe3db864db3aaa15f778c5b007"} Oct 30 22:30:45.474318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:45.474289 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" event={"ID":"df9cec06-7a60-4a6d-a2ea-eea4063436c6","Type":"ContainerStarted","Data":"9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05"} Oct 30 22:30:46.487308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.487242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" event={"ID":"df9cec06-7a60-4a6d-a2ea-eea4063436c6","Type":"ContainerStarted","Data":"f1d1869d44257ccf41ead1c7877ed234dc55db3b905cb8acfa61be7ec2e793b2"} Oct 30 22:30:46.512487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.512416 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" podStartSLOduration=2.51237615 podStartE2EDuration="2.51237615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:46.504247646 +0000 UTC m=+3070.454202593" watchObservedRunningTime="2026-10-30 22:30:46.51237615 +0000 UTC m=+3070.462331119" Oct 30 22:30:46.682294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.682252 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:46.747124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.747032 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\") pod \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " Oct 30 22:30:46.747124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.747092 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\") pod \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " Oct 30 22:30:46.747487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.747456 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca" pod "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" (UID: "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:46.747544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.747477 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\" (UniqueName: \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\") pod \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\" (UID: \"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d\") " Oct 30 22:30:46.748599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.748567 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.752334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.752258 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca" pod "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" (UID: "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:46.752496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.752473 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb" pod "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" (UID: "af65d1cf-0e92-4ad7-8cc2-999c2e47b26d"). InnerVolumeSpecName "kube-api-access-5jxrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:46.850391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.850359 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxrb\" (UniqueName: \"kubernetes.io/projected/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-kube-api-access-5jxrb\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:46.850391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:46.850388 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:47.500649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:47.500612 5036 generic.go:356] "Generic (PLEG): container finished" podID="df9cec06-7a60-4a6d-a2ea-eea4063436c6" containerID="f1d1869d44257ccf41ead1c7877ed234dc55db3b905cb8acfa61be7ec2e793b2" exitCode=0 Oct 30 22:30:47.501074 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:47.500697 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" event={"ID":"df9cec06-7a60-4a6d-a2ea-eea4063436c6","Type":"ContainerDied","Data":"f1d1869d44257ccf41ead1c7877ed234dc55db3b905cb8acfa61be7ec2e793b2"} Oct 30 22:30:47.502432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:47.502413 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" Oct 30 22:30:47.502537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:47.502455 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb" event={"ID":"af65d1cf-0e92-4ad7-8cc2-999c2e47b26d","Type":"ContainerDied","Data":"8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780"} Oct 30 22:30:47.502537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:47.502498 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8508fe9a9afbc6ba0870f1f5ecd7e0f25320e7b9b72818106c716941227ba780" Oct 30 22:30:48.102068 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:30:48.102034 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:30:48.784155 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.784122 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.824754 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\") pod \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.824793 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\" (UniqueName: \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\") pod \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.825108 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\") pod \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\" (UID: \"df9cec06-7a60-4a6d-a2ea-eea4063436c6\") " Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.826284 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca" pod "df9cec06-7a60-4a6d-a2ea-eea4063436c6" (UID: "df9cec06-7a60-4a6d-a2ea-eea4063436c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.828741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk" pod "df9cec06-7a60-4a6d-a2ea-eea4063436c6" (UID: "df9cec06-7a60-4a6d-a2ea-eea4063436c6"). InnerVolumeSpecName "kube-api-access-jgwrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:48.834333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.829318 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca" pod "df9cec06-7a60-4a6d-a2ea-eea4063436c6" (UID: "df9cec06-7a60-4a6d-a2ea-eea4063436c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:48.927741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.927701 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:48.927741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.927735 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9cec06-7a60-4a6d-a2ea-eea4063436c6-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:48.927741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:48.927746 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwrk\" (UniqueName: \"kubernetes.io/projected/df9cec06-7a60-4a6d-a2ea-eea4063436c6-kube-api-access-jgwrk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:49.583128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:49.583095 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" Oct 30 22:30:49.583356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:49.583135 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b" event={"ID":"df9cec06-7a60-4a6d-a2ea-eea4063436c6","Type":"ContainerDied","Data":"9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05"} Oct 30 22:30:49.583356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:49.583216 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9044e36abb030737fabdf52c46df0687accfb1838bce726b015e5ed56b8f1b05" Oct 30 22:30:50.365590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:50.365561 5036 scope.go:122] "RemoveContainer" containerID="bde7ce85755edbe733dee513895686a8cfdee1ab016a0349045ba10fa90061f2" Oct 30 22:30:50.377765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:50.377747 5036 scope.go:122] "RemoveContainer" containerID="9c6b1fb6649c2cc6d5c25985f57093c5aea05c04d984536b75e837698faafd96" Oct 30 22:30:50.391684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:50.391663 5036 scope.go:122] "RemoveContainer" containerID="2ea2ae4dbd725f5f294b440ba272b771bf30eca38c78a80fa232b4f5437d8f6f" Oct 30 22:30:50.402306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:50.402260 5036 scope.go:122] "RemoveContainer" containerID="3d40e7563399c4992cda7bb5c23eb730de63062327ec9407f84873f03102e3b8" Oct 30 22:30:50.413032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:50.413012 5036 scope.go:122] "RemoveContainer" containerID="498612f9f7f4846b959b8eb2aae823af08aa4c66abfe99050e7b5d1c256d1d19" Oct 30 22:30:54.043386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.043100 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct"] Oct 30 22:30:54.044693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.044667 5036 cpu_manager.go:414] "containerMap: removing container" podUID="df9cec06-7a60-4a6d-a2ea-eea4063436c6" containerName="revoke-keys" Oct 30 22:30:54.044693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.044693 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9cec06-7a60-4a6d-a2ea-eea4063436c6" containerName="revoke-keys" Oct 30 22:30:54.044922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.044786 5036 cpu_manager.go:414] "containerMap: removing container" podUID="af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" containerName="revoke-keys" Oct 30 22:30:54.044922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.044798 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" containerName="revoke-keys" Oct 30 22:30:54.045051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.045026 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" containerName="revoke-keys" Oct 30 22:30:54.045128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.045060 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9cec06-7a60-4a6d-a2ea-eea4063436c6" containerName="revoke-keys" Oct 30 22:30:54.052087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.052059 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.059465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.059444 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:30:54.063441 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.063412 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct"] Oct 30 22:30:54.085489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.085460 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:54.116762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.116740 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:54.116905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.116778 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:54.189206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.189170 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.189485 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.189459 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.189541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.189514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsfv\" (UniqueName: \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.291762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.291722 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.292015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.291990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.292107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.292044 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsfv\" (UniqueName: \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.292511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.292480 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.295292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.295218 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.305766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.305742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsfv\" (UniqueName: \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\") pod \"maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.378561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.378533 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:54.775971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:54.775942 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct"] Oct 30 22:30:54.776277 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:30:54.776235 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2c03a7_9298_4e12_b15c_86438cf01c12.slice/crio-f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92 WatchSource:0}: Error finding container f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92: Status 404 returned error can't find the container with id f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92 Oct 30 22:30:55.655736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:55.655701 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" event={"ID":"eb2c03a7-9298-4e12-b15c-86438cf01c12","Type":"ContainerStarted","Data":"21bb86c16190fe194fe376e9bc23e91dbb16b351a53ac984afa1b371a91f78fd"} Oct 30 22:30:55.656284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:55.655753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" event={"ID":"eb2c03a7-9298-4e12-b15c-86438cf01c12","Type":"ContainerStarted","Data":"f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92"} Oct 30 22:30:55.675908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:55.675863 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" podStartSLOduration=1.675849533 podStartE2EDuration="1.675849533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:55.673242464 +0000 UTC m=+3079.623197437" watchObservedRunningTime="2026-10-30 22:30:55.675849533 +0000 UTC m=+3079.625804490" Oct 30 22:30:56.667823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:56.667787 5036 generic.go:356] "Generic (PLEG): container finished" podID="eb2c03a7-9298-4e12-b15c-86438cf01c12" containerID="21bb86c16190fe194fe376e9bc23e91dbb16b351a53ac984afa1b371a91f78fd" exitCode=0 Oct 30 22:30:56.668322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:56.667856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" event={"ID":"eb2c03a7-9298-4e12-b15c-86438cf01c12","Type":"ContainerDied","Data":"21bb86c16190fe194fe376e9bc23e91dbb16b351a53ac984afa1b371a91f78fd"} Oct 30 22:30:57.868695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.868667 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:57.973276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.973182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\" (UniqueName: \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\") pod \"eb2c03a7-9298-4e12-b15c-86438cf01c12\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " Oct 30 22:30:57.973438 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.973312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\") pod \"eb2c03a7-9298-4e12-b15c-86438cf01c12\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " Oct 30 22:30:57.973521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.973504 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\") pod \"eb2c03a7-9298-4e12-b15c-86438cf01c12\" (UID: \"eb2c03a7-9298-4e12-b15c-86438cf01c12\") " Oct 30 22:30:57.974117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.974083 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca" pod "eb2c03a7-9298-4e12-b15c-86438cf01c12" (UID: "eb2c03a7-9298-4e12-b15c-86438cf01c12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:57.974601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.974571 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:57.975717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.975687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv" pod "eb2c03a7-9298-4e12-b15c-86438cf01c12" (UID: "eb2c03a7-9298-4e12-b15c-86438cf01c12"). InnerVolumeSpecName "kube-api-access-kqsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:57.975802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:57.975757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca" pod "eb2c03a7-9298-4e12-b15c-86438cf01c12" (UID: "eb2c03a7-9298-4e12-b15c-86438cf01c12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:58.109291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:58.108976 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsfv\" (UniqueName: \"kubernetes.io/projected/eb2c03a7-9298-4e12-b15c-86438cf01c12-kube-api-access-kqsfv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:58.109291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:58.109014 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2c03a7-9298-4e12-b15c-86438cf01c12-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:58.692217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:58.691324 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" Oct 30 22:30:58.692217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:58.691383 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct" event={"ID":"eb2c03a7-9298-4e12-b15c-86438cf01c12","Type":"ContainerDied","Data":"f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92"} Oct 30 22:30:58.692217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:30:58.691443 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f97cef567561210f4c6189de8b8ca1e008d9e852cd831b0a15011cacdb644a92" Oct 30 22:31:01.376035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.376002 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb"] Oct 30 22:31:01.376916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.376900 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eb2c03a7-9298-4e12-b15c-86438cf01c12" containerName="revoke-keys" Oct 30 22:31:01.376967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.376919 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2c03a7-9298-4e12-b15c-86438cf01c12" containerName="revoke-keys" Oct 30 22:31:01.377071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.377062 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2c03a7-9298-4e12-b15c-86438cf01c12" containerName="revoke-keys" Oct 30 22:31:01.381385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.381366 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.401607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.401579 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:01.401760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.401668 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:31:01.406682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.406659 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb"] Oct 30 22:31:01.422867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.422841 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:01.423022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.422996 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:01.434871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.434835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62zt\" (UniqueName: \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.435047 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.435019 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.435157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.435138 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.537079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.537046 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.537301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.537274 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62zt\" (UniqueName: \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.537375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.537344 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.537819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.537798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.539867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.539848 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.549867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.549840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62zt\" (UniqueName: \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\") pod \"maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:01.697049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:01.697021 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:02.005539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:02.005509 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb"] Oct 30 22:31:02.005984 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:02.005961 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d59eb30_fb9b_46db_a24d_107d64bc6628.slice/crio-d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb WatchSource:0}: Error finding container d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb: Status 404 returned error can't find the container with id d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb Oct 30 22:31:02.737105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:02.737062 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" event={"ID":"2d59eb30-fb9b-46db-a24d-107d64bc6628","Type":"ContainerStarted","Data":"9686d7201c672b7f1877585870511a09463b79f86ac6a972cf8369a0c2ee0387"} Oct 30 22:31:02.737538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:02.737131 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" event={"ID":"2d59eb30-fb9b-46db-a24d-107d64bc6628","Type":"ContainerStarted","Data":"d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb"} Oct 30 22:31:02.755813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:02.755762 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" podStartSLOduration=1.7557498470000001 podStartE2EDuration="1.755749847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:02.754486165 +0000 UTC m=+3086.704441112" watchObservedRunningTime="2026-10-30 22:31:02.755749847 +0000 UTC m=+3086.705704793" Oct 30 22:31:03.751650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:03.751611 5036 generic.go:356] "Generic (PLEG): container finished" podID="2d59eb30-fb9b-46db-a24d-107d64bc6628" containerID="9686d7201c672b7f1877585870511a09463b79f86ac6a972cf8369a0c2ee0387" exitCode=0 Oct 30 22:31:03.752105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:03.751704 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" event={"ID":"2d59eb30-fb9b-46db-a24d-107d64bc6628","Type":"ContainerDied","Data":"9686d7201c672b7f1877585870511a09463b79f86ac6a972cf8369a0c2ee0387"} Oct 30 22:31:04.208352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.208319 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:31:04.233625 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.233593 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:31:04.758378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.758337 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z"] Oct 30 22:31:04.842009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.837399 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:04.861738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.858390 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z"] Oct 30 22:31:04.910870 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.910836 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24lnn\" (UniqueName: \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:04.911690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.911643 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:04.911835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:04.911733 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.035337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.035192 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24lnn\" (UniqueName: \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.035547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.035522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.035713 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.035612 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.036546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.036503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.040301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.040237 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.051922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.051893 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24lnn\" (UniqueName: \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\") pod \"maas-api-revoke-sub-f455a501-2mw9z\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.198284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.195158 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:05.212098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.212069 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:05.342967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.342933 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\") pod \"2d59eb30-fb9b-46db-a24d-107d64bc6628\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " Oct 30 22:31:05.343140 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.343025 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\" (UniqueName: \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\") pod \"2d59eb30-fb9b-46db-a24d-107d64bc6628\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " Oct 30 22:31:05.343199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.343182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\") pod \"2d59eb30-fb9b-46db-a24d-107d64bc6628\" (UID: \"2d59eb30-fb9b-46db-a24d-107d64bc6628\") " Oct 30 22:31:05.344593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.344563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca" pod "2d59eb30-fb9b-46db-a24d-107d64bc6628" (UID: "2d59eb30-fb9b-46db-a24d-107d64bc6628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:05.345226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.345199 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.346086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.346059 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt" pod "2d59eb30-fb9b-46db-a24d-107d64bc6628" (UID: "2d59eb30-fb9b-46db-a24d-107d64bc6628"). InnerVolumeSpecName "kube-api-access-s62zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:05.347062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.347042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca" pod "2d59eb30-fb9b-46db-a24d-107d64bc6628" (UID: "2d59eb30-fb9b-46db-a24d-107d64bc6628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:05.446465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.446433 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62zt\" (UniqueName: \"kubernetes.io/projected/2d59eb30-fb9b-46db-a24d-107d64bc6628-kube-api-access-s62zt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.446465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.446463 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d59eb30-fb9b-46db-a24d-107d64bc6628-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.519892 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.519863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z"] Oct 30 22:31:05.520650 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:05.520619 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a2851_ef4e_40fb_8d75_c94c77fbce3d.slice/crio-de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17 WatchSource:0}: Error finding container de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17: Status 404 returned error can't find the container with id de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17 Oct 30 22:31:05.907591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.907556 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" Oct 30 22:31:05.908081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.907594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb" event={"ID":"2d59eb30-fb9b-46db-a24d-107d64bc6628","Type":"ContainerDied","Data":"d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb"} Oct 30 22:31:05.908081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.907671 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4282be946af780db9d0e6c6823808adda0477d578cd5e161cee7f2ada40e3fb" Oct 30 22:31:05.909576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:05.909538 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" event={"ID":"637a2851-ef4e-40fb-8d75-c94c77fbce3d","Type":"ContainerStarted","Data":"de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17"} Oct 30 22:31:06.922636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:06.922592 5036 generic.go:356] "Generic (PLEG): container finished" podID="637a2851-ef4e-40fb-8d75-c94c77fbce3d" containerID="dc5a72cf1ca80868037a36fe635b43b8a7d32fb492e344e61528775bba97d250" exitCode=0 Oct 30 22:31:06.923120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:06.922674 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" event={"ID":"637a2851-ef4e-40fb-8d75-c94c77fbce3d","Type":"ContainerDied","Data":"dc5a72cf1ca80868037a36fe635b43b8a7d32fb492e344e61528775bba97d250"} Oct 30 22:31:08.128219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.128187 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:08.312105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.311937 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\") pod \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " Oct 30 22:31:08.313106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.313083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\" (UniqueName: \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\") pod \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " Oct 30 22:31:08.313726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.313704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\") pod \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\" (UID: \"637a2851-ef4e-40fb-8d75-c94c77fbce3d\") " Oct 30 22:31:08.315931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.312885 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca" pod "637a2851-ef4e-40fb-8d75-c94c77fbce3d" (UID: "637a2851-ef4e-40fb-8d75-c94c77fbce3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:08.318641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.318603 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca" pod "637a2851-ef4e-40fb-8d75-c94c77fbce3d" (UID: "637a2851-ef4e-40fb-8d75-c94c77fbce3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:08.332163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.324881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn" pod "637a2851-ef4e-40fb-8d75-c94c77fbce3d" (UID: "637a2851-ef4e-40fb-8d75-c94c77fbce3d"). InnerVolumeSpecName "kube-api-access-24lnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:08.417843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.417800 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24lnn\" (UniqueName: \"kubernetes.io/projected/637a2851-ef4e-40fb-8d75-c94c77fbce3d-kube-api-access-24lnn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:08.417843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.417841 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:08.418093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.417856 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637a2851-ef4e-40fb-8d75-c94c77fbce3d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:08.951111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.951077 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" Oct 30 22:31:08.951338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.951083 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z" event={"ID":"637a2851-ef4e-40fb-8d75-c94c77fbce3d","Type":"ContainerDied","Data":"de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17"} Oct 30 22:31:08.951338 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:08.951284 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6d3067a2e30fb24083107d8a4983b7f11ed9ee69f18125d05293b941337e17" Oct 30 22:31:09.208716 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.208605 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" podUID="cfedfbaa-c55d-4483-84db-e74e60da29ff" containerName="payload-processing" containerID="cri-o://d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7" gracePeriod=25 Oct 30 22:31:09.234252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.234213 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" containerID="cri-o://5d7f89d61f808214f9f84138cb069a069bcd1c52d26ffcab25d0093d2f98678a" gracePeriod=25 Oct 30 22:31:09.545621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.545592 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:31:09.640741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.640703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\") pod \"cfedfbaa-c55d-4483-84db-e74e60da29ff\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " Oct 30 22:31:09.641172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.641151 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\" (UniqueName: \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\") pod \"cfedfbaa-c55d-4483-84db-e74e60da29ff\" (UID: \"cfedfbaa-c55d-4483-84db-e74e60da29ff\") " Oct 30 22:31:09.641297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.641152 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume" pod "cfedfbaa-c55d-4483-84db-e74e60da29ff" (UID: "cfedfbaa-c55d-4483-84db-e74e60da29ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:09.642158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.642126 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfedfbaa-c55d-4483-84db-e74e60da29ff-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.643620 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.643595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2" pod "cfedfbaa-c55d-4483-84db-e74e60da29ff" (UID: "cfedfbaa-c55d-4483-84db-e74e60da29ff"). InnerVolumeSpecName "kube-api-access-nh4h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:09.744058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.744018 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4h2\" (UniqueName: \"kubernetes.io/projected/cfedfbaa-c55d-4483-84db-e74e60da29ff-kube-api-access-nh4h2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.988928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.988894 5036 generic.go:356] "Generic (PLEG): container finished" podID="cfedfbaa-c55d-4483-84db-e74e60da29ff" containerID="d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7" exitCode=0 Oct 30 22:31:09.989145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.988941 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" Oct 30 22:31:09.989145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.988974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" event={"ID":"cfedfbaa-c55d-4483-84db-e74e60da29ff","Type":"ContainerDied","Data":"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7"} Oct 30 22:31:09.989145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.989037 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k" event={"ID":"cfedfbaa-c55d-4483-84db-e74e60da29ff","Type":"ContainerDied","Data":"386d0673b47228e0c7e58c4c55508333bddc71168edfe860319c2885ff98b2be"} Oct 30 22:31:09.989145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:09.989068 5036 scope.go:122] "RemoveContainer" containerID="d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7" Oct 30 22:31:10.004804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:10.004782 5036 scope.go:122] "RemoveContainer" containerID="d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7" Oct 30 22:31:10.005144 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:10.005095 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7\": container with ID starting with d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7 not found: ID does not exist" containerID="d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7" Oct 30 22:31:10.005208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:10.005152 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7"} err="failed to get container status \"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7\": rpc error: code = NotFound desc = could not find container \"d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7\": container with ID starting with d5987a8a61a5eea7f2fdf1d1cf1cba5b09c3ff15c7c730dee334482c3de98ce7 not found: ID does not exist" Oct 30 22:31:10.018942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:10.018918 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:31:10.024409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:10.024383 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28d39b-59f545bf7-ldp8k"] Oct 30 22:31:10.190833 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:10.190804 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfedfbaa-c55d-4483-84db-e74e60da29ff" path="/var/lib/kubelet/pods/cfedfbaa-c55d-4483-84db-e74e60da29ff/volumes" Oct 30 22:31:11.627418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:11.627387 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:31:11.662641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:11.662592 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:31:11.798329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:11.798244 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:15.956762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.956679 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk"] Oct 30 22:31:15.957693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957663 5036 cpu_manager.go:414] "containerMap: removing container" podUID="637a2851-ef4e-40fb-8d75-c94c77fbce3d" containerName="revoke-keys" Oct 30 22:31:15.957693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957689 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="637a2851-ef4e-40fb-8d75-c94c77fbce3d" containerName="revoke-keys" Oct 30 22:31:15.957818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957715 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cfedfbaa-c55d-4483-84db-e74e60da29ff" containerName="payload-processing" Oct 30 22:31:15.957818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957724 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfedfbaa-c55d-4483-84db-e74e60da29ff" containerName="payload-processing" Oct 30 22:31:15.957818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957751 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2d59eb30-fb9b-46db-a24d-107d64bc6628" containerName="revoke-keys" Oct 30 22:31:15.957818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957758 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d59eb30-fb9b-46db-a24d-107d64bc6628" containerName="revoke-keys" Oct 30 22:31:15.957995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957964 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="637a2851-ef4e-40fb-8d75-c94c77fbce3d" containerName="revoke-keys" Oct 30 22:31:15.957995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957977 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfedfbaa-c55d-4483-84db-e74e60da29ff" containerName="payload-processing" Oct 30 22:31:15.957995 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.957986 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d59eb30-fb9b-46db-a24d-107d64bc6628" containerName="revoke-keys" Oct 30 22:31:15.963448 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.963426 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:15.994466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.994440 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:15.994668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.994459 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:15.994668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.994481 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:15.994875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.994513 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:31:15.999142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:15.999119 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk"] Oct 30 22:31:16.094300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.094221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.094549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.094527 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9994w\" (UniqueName: \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.094712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.094696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.196426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.196394 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9994w\" (UniqueName: \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.196631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.196545 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.196714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.196644 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.197408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.197379 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.199390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.199225 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.210615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.210559 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9994w\" (UniqueName: \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\") pod \"maas-api-revoke-sub-3cadcd49-2qppk\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.279001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.278961 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:16.633395 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.633290 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" containerID="cri-o://ce3ddc7b4f439cf62a1275cec8468221e5f505a743802ec243b8758d9f043066" gracePeriod=25 Oct 30 22:31:16.662975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.662929 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" containerID="cri-o://2cf668901ce1be06836cd1fcdc64b620a5b429c8de633a9243b118a7accc778b" gracePeriod=25 Oct 30 22:31:16.678889 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:16.678854 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk"] Oct 30 22:31:17.101547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:17.101501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" event={"ID":"36ec71f9-e229-42e7-897e-9f6881c0c1e0","Type":"ContainerStarted","Data":"fc5d83e48e7de8278bc90510771f4aa131ff8d0195cc92c4f17451065fb8b42e"} Oct 30 22:31:18.102872 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:18.102777 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:31:18.113975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:18.113949 5036 generic.go:356] "Generic (PLEG): container finished" podID="36ec71f9-e229-42e7-897e-9f6881c0c1e0" containerID="23792703b93517a1cf0b941a871f38d7a9ba2cfd4c497e0e22ba3883d7b0eb33" exitCode=0 Oct 30 22:31:18.114124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:18.113994 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" event={"ID":"36ec71f9-e229-42e7-897e-9f6881c0c1e0","Type":"ContainerDied","Data":"23792703b93517a1cf0b941a871f38d7a9ba2cfd4c497e0e22ba3883d7b0eb33"} Oct 30 22:31:19.322727 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.322703 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:19.485129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.485101 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\") pod \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " Oct 30 22:31:19.485470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.485443 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\") pod \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " Oct 30 22:31:19.485937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.485914 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\" (UniqueName: \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\") pod \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\" (UID: \"36ec71f9-e229-42e7-897e-9f6881c0c1e0\") " Oct 30 22:31:19.486975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.486005 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca" pod "36ec71f9-e229-42e7-897e-9f6881c0c1e0" (UID: "36ec71f9-e229-42e7-897e-9f6881c0c1e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:19.495969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.495946 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca" pod "36ec71f9-e229-42e7-897e-9f6881c0c1e0" (UID: "36ec71f9-e229-42e7-897e-9f6881c0c1e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:19.497813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.497786 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w" pod "36ec71f9-e229-42e7-897e-9f6881c0c1e0" (UID: "36ec71f9-e229-42e7-897e-9f6881c0c1e0"). InnerVolumeSpecName "kube-api-access-9994w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:19.589168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.589127 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9994w\" (UniqueName: \"kubernetes.io/projected/36ec71f9-e229-42e7-897e-9f6881c0c1e0-kube-api-access-9994w\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:19.589168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.589161 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:19.589372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:19.589174 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ec71f9-e229-42e7-897e-9f6881c0c1e0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:20.138784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:20.138752 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" Oct 30 22:31:20.138784 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:20.138773 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk" event={"ID":"36ec71f9-e229-42e7-897e-9f6881c0c1e0","Type":"ContainerDied","Data":"fc5d83e48e7de8278bc90510771f4aa131ff8d0195cc92c4f17451065fb8b42e"} Oct 30 22:31:20.139058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:20.138818 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5d83e48e7de8278bc90510771f4aa131ff8d0195cc92c4f17451065fb8b42e" Oct 30 22:31:21.541680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:21.541637 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:21.542190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:21.542080 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:21.797725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:21.797642 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:22.390397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:22.390347 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct"] Oct 30 22:31:22.398466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:22.398429 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28d39b-5rnct"] Oct 30 22:31:24.278293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:24.266614 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2c03a7-9298-4e12-b15c-86438cf01c12" path="/var/lib/kubelet/pods/eb2c03a7-9298-4e12-b15c-86438cf01c12/volumes" Oct 30 22:31:24.367649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:24.366483 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb"] Oct 30 22:31:24.388289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:24.385789 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df200a-sr7jb"] Oct 30 22:31:26.193312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.193223 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d59eb30-fb9b-46db-a24d-107d64bc6628" path="/var/lib/kubelet/pods/2d59eb30-fb9b-46db-a24d-107d64bc6628/volumes" Oct 30 22:31:26.849289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.843243 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd"] Oct 30 22:31:26.849289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.844797 5036 cpu_manager.go:414] "containerMap: removing container" podUID="36ec71f9-e229-42e7-897e-9f6881c0c1e0" containerName="revoke-keys" Oct 30 22:31:26.849289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.844820 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ec71f9-e229-42e7-897e-9f6881c0c1e0" containerName="revoke-keys" Oct 30 22:31:26.849289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.845059 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ec71f9-e229-42e7-897e-9f6881c0c1e0" containerName="revoke-keys" Oct 30 22:31:26.863292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.857594 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd"] Oct 30 22:31:26.863292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.857713 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:26.876312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.874949 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:26.900787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.898670 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:31:26.903409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.903151 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:26.903409 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.903238 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:26.903747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.903703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:26.904496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.904087 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:26.904496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:26.904114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvq8b\" (UniqueName: \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.006323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.006278 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.006513 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.006433 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvq8b\" (UniqueName: \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.006723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.006704 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.007138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.007115 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.009284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.009244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.018630 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.018602 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvq8b\" (UniqueName: \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\") pod \"maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.220637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.220607 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:27.525440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:27.525322 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd"] Oct 30 22:31:28.466227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:28.466195 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" event={"ID":"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06","Type":"ContainerStarted","Data":"73329f990a97413a9fce6597691483103c7aae98f0c8932d557919798ab976d4"} Oct 30 22:31:28.466645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:28.466248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" event={"ID":"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06","Type":"ContainerStarted","Data":"2fab64fc2fe1bcb9d36145f5cdcdee639cafa796b031cef39d1f42568eada021"} Oct 30 22:31:28.522260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:28.522213 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" podStartSLOduration=2.522200299 podStartE2EDuration="2.522200299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:28.516927841 +0000 UTC m=+3112.466882788" watchObservedRunningTime="2026-10-30 22:31:28.522200299 +0000 UTC m=+3112.472155244" Oct 30 22:31:28.655950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:28.655913 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:31:28.656221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:28.656193 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" containerID="cri-o://92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94" gracePeriod=30 Oct 30 22:31:29.481386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.481357 5036 generic.go:356] "Generic (PLEG): container finished" podID="b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" containerID="73329f990a97413a9fce6597691483103c7aae98f0c8932d557919798ab976d4" exitCode=0 Oct 30 22:31:29.481845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.481437 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" event={"ID":"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06","Type":"ContainerDied","Data":"73329f990a97413a9fce6597691483103c7aae98f0c8932d557919798ab976d4"} Oct 30 22:31:29.483941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.483914 5036 generic.go:356] "Generic (PLEG): container finished" podID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerID="5d7f89d61f808214f9f84138cb069a069bcd1c52d26ffcab25d0093d2f98678a" exitCode=0 Oct 30 22:31:29.484164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.484148 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" event={"ID":"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb","Type":"ContainerDied","Data":"5d7f89d61f808214f9f84138cb069a069bcd1c52d26ffcab25d0093d2f98678a"} Oct 30 22:31:29.654002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.653972 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:31:29.768726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.768686 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\") pod \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " Oct 30 22:31:29.769016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.768996 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\" (UniqueName: \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\") pod \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\" (UID: \"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb\") " Oct 30 22:31:29.769145 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.769037 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume" pod "9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" (UID: "9c05ac92-b3ae-4da1-b68c-292efe2a3fbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:29.769817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.769798 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.771317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.771290 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj" pod "9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" (UID: "9c05ac92-b3ae-4da1-b68c-292efe2a3fbb"). InnerVolumeSpecName "kube-api-access-gmqzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:29.871873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:29.871793 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqzj\" (UniqueName: \"kubernetes.io/projected/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb-kube-api-access-gmqzj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:30.496397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:30.496307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" event={"ID":"9c05ac92-b3ae-4da1-b68c-292efe2a3fbb","Type":"ContainerDied","Data":"f87041fad1e30e4e327cb45597a8202a94a1d0c776260d32a86c31a8783e4d9a"} Oct 30 22:31:30.496397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:30.496381 5036 scope.go:122] "RemoveContainer" containerID="5d7f89d61f808214f9f84138cb069a069bcd1c52d26ffcab25d0093d2f98678a" Oct 30 22:31:30.496884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:30.496319 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r" Oct 30 22:31:30.602291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:30.600338 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:31:30.732310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:30.713289 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28d39b-8c7db846c-v286r"] Oct 30 22:31:31.069419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.069387 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:31.111702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.111666 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\") pod \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " Oct 30 22:31:31.111888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.111763 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\") pod \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " Oct 30 22:31:31.112076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.112056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\" (UniqueName: \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\") pod \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\" (UID: \"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06\") " Oct 30 22:31:31.112852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.112825 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca" pod "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" (UID: "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:31.115768 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.115739 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca" pod "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" (UID: "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:31.116550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.116514 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b" pod "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" (UID: "b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06"). InnerVolumeSpecName "kube-api-access-gvq8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:31.219333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.218636 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvq8b\" (UniqueName: \"kubernetes.io/projected/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-kube-api-access-gvq8b\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:31.219333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.218675 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:31.219333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.218691 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:31.403453 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.403413 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:31:31.403753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.403720 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" containerID="cri-o://ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9" gracePeriod=30 Oct 30 22:31:31.641289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.637653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:31.641289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.639831 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:31.680326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.668728 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" event={"ID":"b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06","Type":"ContainerDied","Data":"2fab64fc2fe1bcb9d36145f5cdcdee639cafa796b031cef39d1f42568eada021"} Oct 30 22:31:31.680326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.668790 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fab64fc2fe1bcb9d36145f5cdcdee639cafa796b031cef39d1f42568eada021" Oct 30 22:31:31.680326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.668873 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd" Oct 30 22:31:31.861162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.861126 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr"] Oct 30 22:31:31.862071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862045 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" containerName="revoke-keys" Oct 30 22:31:31.862071 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862070 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" containerName="revoke-keys" Oct 30 22:31:31.862250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862114 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" Oct 30 22:31:31.862250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862124 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" Oct 30 22:31:31.862546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862529 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" containerName="revoke-keys" Oct 30 22:31:31.862605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.862561 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" containerName="payload-pre-processing" Oct 30 22:31:31.866791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.866772 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:31.879631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.879608 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:31.880025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.879916 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:31.880227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.880213 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:31:31.880463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.880448 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:31.884761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.884738 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr"] Oct 30 22:31:31.951936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.951845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:31.952112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.952034 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfdt\" (UniqueName: \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:31.952285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:31.952243 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.053959 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.053916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.054154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.054082 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfdt\" (UniqueName: \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.054335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.054310 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.054743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.054719 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.057118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.057085 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.070087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.070060 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfdt\" (UniqueName: \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\") pod \"maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.186969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.186934 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:32.194176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.194146 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c05ac92-b3ae-4da1-b68c-292efe2a3fbb" path="/var/lib/kubelet/pods/9c05ac92-b3ae-4da1-b68c-292efe2a3fbb/volumes" Oct 30 22:31:32.584149 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:32.584098 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84df9f_6473_4a03_a8e3_44f5ec9f52a4.slice/crio-f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9 WatchSource:0}: Error finding container f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9: Status 404 returned error can't find the container with id f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9 Oct 30 22:31:32.586061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.586031 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr"] Oct 30 22:31:32.795290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:32.795179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" event={"ID":"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4","Type":"ContainerStarted","Data":"f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9"} Oct 30 22:31:33.202333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.120722 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:31:33.202333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.121011 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" containerID="cri-o://e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386" gracePeriod=30 Oct 30 22:31:33.257364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.255310 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:31:33.257364 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.255561 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" containerID="cri-o://24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787" gracePeriod=30 Oct 30 22:31:33.763288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.760326 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq"] Oct 30 22:31:33.789336 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.771681 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:33.825547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.824440 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" event={"ID":"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4","Type":"ContainerStarted","Data":"c59f5223dee24488554c532e8ea87331026baecc3cc4f43c0681b615a86080e1"} Oct 30 22:31:33.859549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.850088 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq"] Oct 30 22:31:33.861459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.861409 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" podStartSLOduration=2.861394295 podStartE2EDuration="2.861394295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:33.84264366 +0000 UTC m=+3117.792598607" watchObservedRunningTime="2026-10-30 22:31:33.861394295 +0000 UTC m=+3117.811349280" Oct 30 22:31:33.927296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.923214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:33.927296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.923410 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:33.927296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:33.923609 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwb6d\" (UniqueName: \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.037310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.027592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.037310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.027714 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.037310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.027874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwb6d\" (UniqueName: \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.037310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.029458 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.044726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.044659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.075704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.075633 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwb6d\" (UniqueName: \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\") pod \"maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.166289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.151796 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:34.428739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.428711 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:31:34.550160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550077 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.550160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550135 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.550450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550326 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.550450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550444 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.551028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550546 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.551028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.551028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550620 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.551028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550811 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.551028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550933 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:34.551955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.550852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\") pod \"8c2dfe1f-6599-40a0-9369-f7f720c90647\" (UID: \"8c2dfe1f-6599-40a0-9369-f7f720c90647\") " Oct 30 22:31:34.552760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.552694 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:34.552760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.552709 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:34.552760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.552710 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.553561 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.553683 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "kube-api-access-69rgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554516 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rgg\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-kube-api-access-69rgg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554543 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554559 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2dfe1f-6599-40a0-9369-f7f720c90647-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554580 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554594 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.554609 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.560568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.555132 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:34.561421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.560793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:34.564312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.562826 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq"] Oct 30 22:31:34.634787 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:34.634727 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3b32e3_67c2_4f60_813b_aca0436d24b1.slice/crio-b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2 WatchSource:0}: Error finding container b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2: Status 404 returned error can't find the container with id b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2 Oct 30 22:31:34.653240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.653191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy" pod "8c2dfe1f-6599-40a0-9369-f7f720c90647" (UID: "8c2dfe1f-6599-40a0-9369-f7f720c90647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:34.657874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.657415 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.657874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.657460 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.657874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.657476 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2dfe1f-6599-40a0-9369-f7f720c90647-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:34.835986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.835902 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" event={"ID":"db3b32e3-67c2-4f60-813b-aca0436d24b1","Type":"ContainerStarted","Data":"b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2"} Oct 30 22:31:34.837827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.837795 5036 generic.go:356] "Generic (PLEG): container finished" podID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerID="92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94" exitCode=0 Oct 30 22:31:34.837983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.837865 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" Oct 30 22:31:34.837983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.837886 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" event={"ID":"8c2dfe1f-6599-40a0-9369-f7f720c90647","Type":"ContainerDied","Data":"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94"} Oct 30 22:31:34.837983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.837969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z" event={"ID":"8c2dfe1f-6599-40a0-9369-f7f720c90647","Type":"ContainerDied","Data":"ad2adf394a4168cd6a43f16ba4185d382bda24e1ee13318f54dfd39d91c9aa37"} Oct 30 22:31:34.838144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.838002 5036 scope.go:122] "RemoveContainer" containerID="92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94" Oct 30 22:31:34.839943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.839916 5036 generic.go:356] "Generic (PLEG): container finished" podID="7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" containerID="c59f5223dee24488554c532e8ea87331026baecc3cc4f43c0681b615a86080e1" exitCode=0 Oct 30 22:31:34.840052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.839969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" event={"ID":"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4","Type":"ContainerDied","Data":"c59f5223dee24488554c532e8ea87331026baecc3cc4f43c0681b615a86080e1"} Oct 30 22:31:34.854428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.854411 5036 scope.go:122] "RemoveContainer" containerID="92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94" Oct 30 22:31:34.854731 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:34.854707 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94\": container with ID starting with 92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94 not found: ID does not exist" containerID="92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94" Oct 30 22:31:34.854851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.854744 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94"} err="failed to get container status \"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94\": rpc error: code = NotFound desc = could not find container \"92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94\": container with ID starting with 92821d461d6baece853d25a0437325652402caed17ccc7a49594c98a70677c94 not found: ID does not exist" Oct 30 22:31:34.883482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.883412 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:31:34.889476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:34.889445 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-w7g8z"] Oct 30 22:31:35.853451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:35.853412 5036 generic.go:356] "Generic (PLEG): container finished" podID="db3b32e3-67c2-4f60-813b-aca0436d24b1" containerID="583d590f58705fd0d59431f2c1cffe4d3b27f3c6b64d368a19fc0e5f1d70e708" exitCode=0 Oct 30 22:31:35.853931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:35.853680 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" event={"ID":"db3b32e3-67c2-4f60-813b-aca0436d24b1","Type":"ContainerDied","Data":"583d590f58705fd0d59431f2c1cffe4d3b27f3c6b64d368a19fc0e5f1d70e708"} Oct 30 22:31:36.049644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.049606 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:36.205596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.205503 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\" (UniqueName: \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\") pod \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " Oct 30 22:31:36.205778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.205603 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\") pod \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " Oct 30 22:31:36.205778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.205673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\") pod \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\" (UID: \"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4\") " Oct 30 22:31:36.225412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.225257 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca" pod "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" (UID: "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:36.234158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.234111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt" pod "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" (UID: "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4"). InnerVolumeSpecName "kube-api-access-dtfdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.241522 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.241464 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca" pod "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" (UID: "7e84df9f-6473-4a03-a8e3-44f5ec9f52a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:36.249049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.249017 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" path="/var/lib/kubelet/pods/8c2dfe1f-6599-40a0-9369-f7f720c90647/volumes" Oct 30 22:31:36.307831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.307801 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfdt\" (UniqueName: \"kubernetes.io/projected/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-kube-api-access-dtfdt\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.307831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.307827 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.308035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.307839 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.781016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.780985 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:31:36.926643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.926611 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.930684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.930658 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.931370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.931346 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.931666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.931649 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.937017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.936992 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.938256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.938233 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.938444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.938425 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.938658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.938637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.939193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.939172 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\") pod \"f98fec20-adfa-4090-a5ed-39d187302155\" (UID: \"f98fec20-adfa-4090-a5ed-39d187302155\") " Oct 30 22:31:36.946020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.945998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:36.946631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.946571 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.947092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.947066 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:36.947402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.947377 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.947634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.947593 5036 generic.go:356] "Generic (PLEG): container finished" podID="f98fec20-adfa-4090-a5ed-39d187302155" containerID="ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9" exitCode=0 Oct 30 22:31:36.948118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.948056 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" Oct 30 22:31:36.950333 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.949815 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "kube-api-access-g6xf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.950436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.950017 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.950436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.950094 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.950436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.949839 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerDied","Data":"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9"} Oct 30 22:31:36.950436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.950403 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj" event={"ID":"f98fec20-adfa-4090-a5ed-39d187302155","Type":"ContainerDied","Data":"5c83203cede0f6f6aa4db7051bde5a87ef5615ab2daa4c1d36c2bbbefe507b40"} Oct 30 22:31:36.950672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.950437 5036 scope.go:122] "RemoveContainer" containerID="ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9" Oct 30 22:31:36.953972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.953672 5036 generic.go:356] "Generic (PLEG): container finished" podID="be41384f-2bc1-4094-a410-6aa343541589" containerID="ce3ddc7b4f439cf62a1275cec8468221e5f505a743802ec243b8758d9f043066" exitCode=0 Oct 30 22:31:36.953972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.953882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" event={"ID":"be41384f-2bc1-4094-a410-6aa343541589","Type":"ContainerDied","Data":"ce3ddc7b4f439cf62a1275cec8468221e5f505a743802ec243b8758d9f043066"} Oct 30 22:31:36.956778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.956703 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" event={"ID":"7e84df9f-6473-4a03-a8e3-44f5ec9f52a4","Type":"ContainerDied","Data":"f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9"} Oct 30 22:31:36.956778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.956745 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e60411ef09f140120383a4eaebcd34b449190b4fb3fc96084518b7fc514dd9" Oct 30 22:31:36.957021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.956784 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr" Oct 30 22:31:36.959855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.959552 5036 generic.go:356] "Generic (PLEG): container finished" podID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerID="2cf668901ce1be06836cd1fcdc64b620a5b429c8de633a9243b118a7accc778b" exitCode=0 Oct 30 22:31:36.959855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.959796 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" event={"ID":"3d657993-9a1e-4f43-b331-7d758a6245ab","Type":"ContainerDied","Data":"2cf668901ce1be06836cd1fcdc64b620a5b429c8de633a9243b118a7accc778b"} Oct 30 22:31:36.971628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.970613 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.972445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.972419 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:31:36.973638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.973561 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token" pod "f98fec20-adfa-4090-a5ed-39d187302155" (UID: "f98fec20-adfa-4090-a5ed-39d187302155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.998551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:36.998519 5036 scope.go:122] "RemoveContainer" containerID="007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95" Oct 30 22:31:37.006598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.004582 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:31:37.026060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.026029 5036 scope.go:122] "RemoveContainer" containerID="ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9" Oct 30 22:31:37.026692 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:37.026653 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9\": container with ID starting with ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9 not found: ID does not exist" containerID="ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9" Oct 30 22:31:37.026808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.026704 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9"} err="failed to get container status \"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9\": rpc error: code = NotFound desc = could not find container \"ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9\": container with ID starting with ca41dc3d181485aab916016dc77dc361b9574533076053c8799afaf670d016c9 not found: ID does not exist" Oct 30 22:31:37.026808 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.026730 5036 scope.go:122] "RemoveContainer" containerID="007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95" Oct 30 22:31:37.029872 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:37.029721 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95\": container with ID starting with 007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95 not found: ID does not exist" containerID="007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95" Oct 30 22:31:37.030054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.030022 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95"} err="failed to get container status \"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95\": rpc error: code = NotFound desc = could not find container \"007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95\": container with ID starting with 007501e9e47649f1892118d6d4361197d7bff1857130c0c16eac0641323ebf95 not found: ID does not exist" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.043942 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98fec20-adfa-4090-a5ed-39d187302155-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.043989 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044007 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044032 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044050 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6xf4\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-kube-api-access-g6xf4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044067 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044083 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98fec20-adfa-4090-a5ed-39d187302155-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044102 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98fec20-adfa-4090-a5ed-39d187302155-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.044373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.044117 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98fec20-adfa-4090-a5ed-39d187302155-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.564541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.564511 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:31:37.573164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.573106 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-9sptj"] Oct 30 22:31:37.605539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.605511 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:31:37.672319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.668128 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:31:37.678881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.678850 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:37.707847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.707813 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\" (UniqueName: \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\") pod \"be41384f-2bc1-4094-a410-6aa343541589\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " Oct 30 22:31:37.708553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.708526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\") pod \"3d657993-9a1e-4f43-b331-7d758a6245ab\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " Oct 30 22:31:37.708786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.708768 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\" (UniqueName: \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\") pod \"3d657993-9a1e-4f43-b331-7d758a6245ab\" (UID: \"3d657993-9a1e-4f43-b331-7d758a6245ab\") " Oct 30 22:31:37.709093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.709076 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\") pod \"be41384f-2bc1-4094-a410-6aa343541589\" (UID: \"be41384f-2bc1-4094-a410-6aa343541589\") " Oct 30 22:31:37.712362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.712292 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume" pod "be41384f-2bc1-4094-a410-6aa343541589" (UID: "be41384f-2bc1-4094-a410-6aa343541589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:37.714050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.714015 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume" pod "3d657993-9a1e-4f43-b331-7d758a6245ab" (UID: "3d657993-9a1e-4f43-b331-7d758a6245ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:37.714507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.714438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg" pod "be41384f-2bc1-4094-a410-6aa343541589" (UID: "be41384f-2bc1-4094-a410-6aa343541589"). InnerVolumeSpecName "kube-api-access-l9lpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:37.714815 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.714788 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d657993-9a1e-4f43-b331-7d758a6245ab-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.715703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.714940 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be41384f-2bc1-4094-a410-6aa343541589-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.716024 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.715976 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h" pod "3d657993-9a1e-4f43-b331-7d758a6245ab" (UID: "3d657993-9a1e-4f43-b331-7d758a6245ab"). InnerVolumeSpecName "kube-api-access-zdg6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:37.817142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.817108 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\" (UniqueName: \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\") pod \"db3b32e3-67c2-4f60-813b-aca0436d24b1\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " Oct 30 22:31:37.817322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.817250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\") pod \"db3b32e3-67c2-4f60-813b-aca0436d24b1\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " Oct 30 22:31:37.817411 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.817363 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\") pod \"db3b32e3-67c2-4f60-813b-aca0436d24b1\" (UID: \"db3b32e3-67c2-4f60-813b-aca0436d24b1\") " Oct 30 22:31:37.817776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.817748 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca" pod "db3b32e3-67c2-4f60-813b-aca0436d24b1" (UID: "db3b32e3-67c2-4f60-813b-aca0436d24b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:37.818083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.818064 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.818154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.818086 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lpg\" (UniqueName: \"kubernetes.io/projected/be41384f-2bc1-4094-a410-6aa343541589-kube-api-access-l9lpg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.818154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.818096 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdg6h\" (UniqueName: \"kubernetes.io/projected/3d657993-9a1e-4f43-b331-7d758a6245ab-kube-api-access-zdg6h\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.819514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.819492 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d" pod "db3b32e3-67c2-4f60-813b-aca0436d24b1" (UID: "db3b32e3-67c2-4f60-813b-aca0436d24b1"). InnerVolumeSpecName "kube-api-access-cwb6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:37.819514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.819498 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca" pod "db3b32e3-67c2-4f60-813b-aca0436d24b1" (UID: "db3b32e3-67c2-4f60-813b-aca0436d24b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:37.919992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.919962 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwb6d\" (UniqueName: \"kubernetes.io/projected/db3b32e3-67c2-4f60-813b-aca0436d24b1-kube-api-access-cwb6d\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.919992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.919991 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3b32e3-67c2-4f60-813b-aca0436d24b1-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.973241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.973215 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" Oct 30 22:31:37.973666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.973237 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6" event={"ID":"3d657993-9a1e-4f43-b331-7d758a6245ab","Type":"ContainerDied","Data":"e6dae9d49e944a933019c791651b14568598f9a52a23ec45105f0efa17676276"} Oct 30 22:31:37.973666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.973322 5036 scope.go:122] "RemoveContainer" containerID="2cf668901ce1be06836cd1fcdc64b620a5b429c8de633a9243b118a7accc778b" Oct 30 22:31:37.976591 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.976562 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" event={"ID":"db3b32e3-67c2-4f60-813b-aca0436d24b1","Type":"ContainerDied","Data":"b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2"} Oct 30 22:31:37.976700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.976614 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28e68e7d7d912b6bb02ed53a701c8acddd2f8788c0b422b141d8b93043454b2" Oct 30 22:31:37.976700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.976671 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq" Oct 30 22:31:37.983225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.983198 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" event={"ID":"be41384f-2bc1-4094-a410-6aa343541589","Type":"ContainerDied","Data":"6f171725a7361c6b8460ad39d1d3e681b260f02446e7714f5e1d2721581ab64c"} Oct 30 22:31:37.983389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.983232 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5" Oct 30 22:31:37.987610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:37.987586 5036 scope.go:122] "RemoveContainer" containerID="ce3ddc7b4f439cf62a1275cec8468221e5f505a743802ec243b8758d9f043066" Oct 30 22:31:38.005087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.005067 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:31:38.011988 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.011961 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df200a-b4f858b79-p46f6"] Oct 30 22:31:38.024853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.024826 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:31:38.030410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.030387 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df200a-64bcccc67f-7j5j5"] Oct 30 22:31:38.190577 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.190496 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" path="/var/lib/kubelet/pods/3d657993-9a1e-4f43-b331-7d758a6245ab/volumes" Oct 30 22:31:38.191113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.191093 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be41384f-2bc1-4094-a410-6aa343541589" path="/var/lib/kubelet/pods/be41384f-2bc1-4094-a410-6aa343541589/volumes" Oct 30 22:31:38.191747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.191722 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98fec20-adfa-4090-a5ed-39d187302155" path="/var/lib/kubelet/pods/f98fec20-adfa-4090-a5ed-39d187302155/volumes" Oct 30 22:31:38.493039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.492944 5036 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body= Oct 30 22:31:38.493039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.493006 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" Oct 30 22:31:38.651563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.651530 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:31:38.724076 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.724050 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752090 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752652 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.752825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752689 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.753311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752846 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.753311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.752898 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\") pod \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\" (UID: \"75903261-86d8-4087-9fa3-6bc5e1d92e8c\") " Oct 30 22:31:38.755374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.755092 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.755769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.755747 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.756342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.756319 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.759296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.759252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.760084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.760048 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "kube-api-access-scln4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.760423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.760402 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:38.762602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.762577 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.762733 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.762686 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.764009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.763980 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo" pod "75903261-86d8-4087-9fa3-6bc5e1d92e8c" (UID: "75903261-86d8-4087-9fa3-6bc5e1d92e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:38.854350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854307 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.854551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854484 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.854551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854539 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.854676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854624 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.854740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854675 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.854869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854847 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.855096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.855076 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.855388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854876 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.855388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.854898 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.855388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.855236 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:38.855823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.855774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.855946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.855848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\") pod \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\" (UID: \"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade\") " Oct 30 22:31:38.856429 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856402 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.856822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856800 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856827 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856842 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856854 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856868 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856882 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856894 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856912 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856925 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856939 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856955 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.856967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856968 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75903261-86d8-4087-9fa3-6bc5e1d92e8c-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.857544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.856983 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scln4\" (UniqueName: \"kubernetes.io/projected/75903261-86d8-4087-9fa3-6bc5e1d92e8c-kube-api-access-scln4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.857544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.857126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.857544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.857194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "kube-api-access-7fpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.857544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.857441 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.857544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.857492 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.857775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.857758 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo" pod "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" (UID: "6a83fd3d-d397-4f39-b2cc-8ae7feec6ade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:38.958891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.958856 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fpfz\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-kube-api-access-7fpfz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.958891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.958890 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.959122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.958902 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.959122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.958916 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.959122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.958929 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.995391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.995357 5036 generic.go:356] "Generic (PLEG): container finished" podID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerID="e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386" exitCode=0 Oct 30 22:31:38.995831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.995410 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" Oct 30 22:31:38.995831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.995451 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerDied","Data":"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386"} Oct 30 22:31:38.995831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.995516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b" event={"ID":"6a83fd3d-d397-4f39-b2cc-8ae7feec6ade","Type":"ContainerDied","Data":"6211f3512ccacda2a4805f7204e441efa77afaa4d38f0c8b2992ed104f214ee9"} Oct 30 22:31:38.995831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.995545 5036 scope.go:122] "RemoveContainer" containerID="e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386" Oct 30 22:31:38.998813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.998787 5036 generic.go:356] "Generic (PLEG): container finished" podID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerID="24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787" exitCode=0 Oct 30 22:31:38.998957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.998832 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" Oct 30 22:31:38.998957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.998895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" event={"ID":"75903261-86d8-4087-9fa3-6bc5e1d92e8c","Type":"ContainerDied","Data":"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787"} Oct 30 22:31:38.999083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:38.999002 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs" event={"ID":"75903261-86d8-4087-9fa3-6bc5e1d92e8c","Type":"ContainerDied","Data":"b057638d8557fd2205bab653c16e18f627baf10417ac42e2274bd19fceac7a6c"} Oct 30 22:31:39.019476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.019455 5036 scope.go:122] "RemoveContainer" containerID="a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354" Oct 30 22:31:39.025349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.025324 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:31:39.031589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.031566 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-6hj2b"] Oct 30 22:31:39.033854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.033830 5036 scope.go:122] "RemoveContainer" containerID="e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386" Oct 30 22:31:39.034216 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:39.034185 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386\": container with ID starting with e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386 not found: ID does not exist" containerID="e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386" Oct 30 22:31:39.034390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.034235 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386"} err="failed to get container status \"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386\": rpc error: code = NotFound desc = could not find container \"e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386\": container with ID starting with e2813ce6da5de986e153749ac6c310d12acda35a790f23253dd040187aa33386 not found: ID does not exist" Oct 30 22:31:39.034390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.034305 5036 scope.go:122] "RemoveContainer" containerID="a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354" Oct 30 22:31:39.034729 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:39.034697 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354\": container with ID starting with a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354 not found: ID does not exist" containerID="a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354" Oct 30 22:31:39.034729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.034721 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354"} err="failed to get container status \"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354\": rpc error: code = NotFound desc = could not find container \"a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354\": container with ID starting with a72b1e07eeaefe7682dcccf93a4dd8dbf89002f8d182cdb7fcf763b3d8dc9354 not found: ID does not exist" Oct 30 22:31:39.034883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.034737 5036 scope.go:122] "RemoveContainer" containerID="24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787" Oct 30 22:31:39.042369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.042335 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:31:39.045937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.045911 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cgzvs"] Oct 30 22:31:39.048464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.048432 5036 scope.go:122] "RemoveContainer" containerID="24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787" Oct 30 22:31:39.048842 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:39.048823 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787\": container with ID starting with 24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787 not found: ID does not exist" containerID="24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787" Oct 30 22:31:39.048921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:39.048848 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787"} err="failed to get container status \"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787\": rpc error: code = NotFound desc = could not find container \"24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787\": container with ID starting with 24cbf282514c9fb64e740b37453fc38027c8704fc28d3cb9d5dc964e414af787 not found: ID does not exist" Oct 30 22:31:40.191832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:40.191803 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" path="/var/lib/kubelet/pods/6a83fd3d-d397-4f39-b2cc-8ae7feec6ade/volumes" Oct 30 22:31:40.192363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:40.192345 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" path="/var/lib/kubelet/pods/75903261-86d8-4087-9fa3-6bc5e1d92e8c/volumes" Oct 30 22:31:41.975402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:41.975341 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" podUID="70097ceb-7814-42d1-a94a-d269decc53ac" containerName="payload-processing" containerID="cri-o://63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe" gracePeriod=25 Oct 30 22:31:42.005729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.005663 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" containerID="cri-o://636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190" gracePeriod=25 Oct 30 22:31:42.007591 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:42.007110 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70097ceb_7814_42d1_a94a_d269decc53ac.slice/crio-63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:31:42.632373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.632224 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:31:42.786799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.786767 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\") pod \"70097ceb-7814-42d1-a94a-d269decc53ac\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " Oct 30 22:31:42.787012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.786838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\" (UniqueName: \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\") pod \"70097ceb-7814-42d1-a94a-d269decc53ac\" (UID: \"70097ceb-7814-42d1-a94a-d269decc53ac\") " Oct 30 22:31:42.787184 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.787155 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume" pod "70097ceb-7814-42d1-a94a-d269decc53ac" (UID: "70097ceb-7814-42d1-a94a-d269decc53ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:42.787992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.787958 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70097ceb-7814-42d1-a94a-d269decc53ac-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.789440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.789405 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx" pod "70097ceb-7814-42d1-a94a-d269decc53ac" (UID: "70097ceb-7814-42d1-a94a-d269decc53ac"). InnerVolumeSpecName "kube-api-access-sf9gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:42.889251 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:42.889222 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9gx\" (UniqueName: \"kubernetes.io/projected/70097ceb-7814-42d1-a94a-d269decc53ac-kube-api-access-sf9gx\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.102421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.102384 5036 generic.go:356] "Generic (PLEG): container finished" podID="70097ceb-7814-42d1-a94a-d269decc53ac" containerID="63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe" exitCode=0 Oct 30 22:31:43.102829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.102427 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" event={"ID":"70097ceb-7814-42d1-a94a-d269decc53ac","Type":"ContainerDied","Data":"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe"} Oct 30 22:31:43.102829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.102434 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" Oct 30 22:31:43.102829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.102475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr" event={"ID":"70097ceb-7814-42d1-a94a-d269decc53ac","Type":"ContainerDied","Data":"c1bf1ea34764eac508d0fb4c0f127e0c120b21976ca614ddd7f1da4658e36c9e"} Oct 30 22:31:43.102829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.102500 5036 scope.go:122] "RemoveContainer" containerID="63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe" Oct 30 22:31:43.117454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.117428 5036 scope.go:122] "RemoveContainer" containerID="63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe" Oct 30 22:31:43.117834 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:43.117800 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe\": container with ID starting with 63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe not found: ID does not exist" containerID="63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe" Oct 30 22:31:43.117944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.117843 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe"} err="failed to get container status \"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe\": rpc error: code = NotFound desc = could not find container \"63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe\": container with ID starting with 63e2a2901af5bf9cd249a4eda8728566b2d330f3c41964108bcb371a1a812dfe not found: ID does not exist" Oct 30 22:31:43.135310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.135251 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:31:43.153112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.153018 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-120cb1-76d8cb9f9f-52pnr"] Oct 30 22:31:43.655292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.654460 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd"] Oct 30 22:31:43.662861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:43.662821 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-120cb1-sclfd"] Oct 30 22:31:44.232292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:44.231970 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70097ceb-7814-42d1-a94a-d269decc53ac" path="/var/lib/kubelet/pods/70097ceb-7814-42d1-a94a-d269decc53ac/volumes" Oct 30 22:31:44.251868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:44.251313 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06" path="/var/lib/kubelet/pods/b2f7f01d-07f1-4c8f-acde-9c0a5dc02b06/volumes" Oct 30 22:31:47.854089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:47.854054 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr"] Oct 30 22:31:47.872374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:47.872299 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d5fa-z2rfr"] Oct 30 22:31:48.111548 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:31:48.111423 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:31:48.190610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:48.190579 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" path="/var/lib/kubelet/pods/7e84df9f-6473-4a03-a8e3-44f5ec9f52a4/volumes" Oct 30 22:31:50.526908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:50.526865 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:31:50.591144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:50.591107 5036 scope.go:122] "RemoveContainer" containerID="91e42016320b7b55ef060bcf74703f2873c702e2432385254c48f6d77dab48e8" Oct 30 22:31:50.602801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:50.602780 5036 scope.go:122] "RemoveContainer" containerID="b3a98d6a65766ef2dd0ba47b61ae2d945044f8c7bf66470b9e357a7a979b798e" Oct 30 22:31:50.613863 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:50.613837 5036 scope.go:122] "RemoveContainer" containerID="a72a20d55c075feea1f3fbaeeb35a4c73038f1e2e8d6a09320f13238c8abfcbb" Oct 30 22:31:50.627385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:50.627351 5036 scope.go:122] "RemoveContainer" containerID="4b663830632ebfb00b15b08437b0605ccec0e131709da4217b4bbb13a52c878a" Oct 30 22:31:51.145871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.145830 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:31:51.146543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.146180 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" containerID="cri-o://cecdfb7226fca9d79144767e66f0812ea60298c335980a91e809478a4fb4792c" gracePeriod=30 Oct 30 22:31:51.154253 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.154161 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:31:51.154648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.154610 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" containerID="cri-o://4685d4b9acb233f3312f6023502ac733a528c01ab7006ee14cfcdd7fdb31a9d6" gracePeriod=30 Oct 30 22:31:51.165070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.165040 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:31:51.165467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.165437 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" containerID="cri-o://2ec1b170c6fd2a13a2efe178f45543f2e0d5044c131a199d57634cdde3b2b417" gracePeriod=30 Oct 30 22:31:51.180939 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.180904 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:31:51.181141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.181119 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" containerID="cri-o://5a64d815843df9d12cf808d95b3cff3be503ce058bb8ac284af3c81a99d46475" gracePeriod=30 Oct 30 22:31:51.492722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.492542 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:51.492722 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.492603 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:51.694150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.694106 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7"] Oct 30 22:31:51.695406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695384 5036 cpu_manager.go:414] "containerMap: removing container" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" Oct 30 22:31:51.695482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695411 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" Oct 30 22:31:51.695482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695467 5036 cpu_manager.go:414] "containerMap: removing container" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" Oct 30 22:31:51.695482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695475 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695506 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" containerName="revoke-keys" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695515 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" containerName="revoke-keys" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695528 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695536 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695559 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695566 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695579 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.695598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695586 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695603 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70097ceb-7814-42d1-a94a-d269decc53ac" containerName="payload-processing" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695610 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70097ceb-7814-42d1-a94a-d269decc53ac" containerName="payload-processing" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695641 5036 cpu_manager.go:414] "containerMap: removing container" podUID="db3b32e3-67c2-4f60-813b-aca0436d24b1" containerName="revoke-keys" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695649 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3b32e3-67c2-4f60-813b-aca0436d24b1" containerName="revoke-keys" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695674 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695681 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695703 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695724 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695735 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.695841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695742 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695937 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84df9f-6473-4a03-a8e3-44f5ec9f52a4" containerName="revoke-keys" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695952 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d657993-9a1e-4f43-b331-7d758a6245ab" containerName="payload-pre-processing" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695965 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2dfe1f-6599-40a0-9369-f7f720c90647" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695975 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695985 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70097ceb-7814-42d1-a94a-d269decc53ac" containerName="payload-processing" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.695999 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3b32e3-67c2-4f60-813b-aca0436d24b1" containerName="revoke-keys" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.696012 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98fec20-adfa-4090-a5ed-39d187302155" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.696024 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.696034 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a83fd3d-d397-4f39-b2cc-8ae7feec6ade" containerName="istio-proxy" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.696047 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="be41384f-2bc1-4094-a410-6aa343541589" containerName="payload-processing" Oct 30 22:31:51.696132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.696062 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75903261-86d8-4087-9fa3-6bc5e1d92e8c" containerName="istio-proxy" Oct 30 22:31:51.703072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.703047 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.716612 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.716583 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:31:51.716895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.716877 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:31:51.717120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.716636 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:51.717288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.717256 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:31:51.717763 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.717736 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7"] Oct 30 22:31:51.821033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.820945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hfk\" (UniqueName: \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.821190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.821170 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.821277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.821238 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.922795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.922756 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.922978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.922868 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.923102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.923072 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hfk\" (UniqueName: \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.925818 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.924869 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:51.927291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.926799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:52.004288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:51.999443 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hfk\" (UniqueName: \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\") pod \"maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:52.048295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.038313 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:52.288917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.288883 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq"] Oct 30 22:31:52.297034 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.297010 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c92aec-gs4kq"] Oct 30 22:31:52.419929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.418424 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:52.419929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.418486 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:52.613837 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:52.613808 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c546161_cc87_4a97_8101_532ce6ae58f9.slice/crio-ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835 WatchSource:0}: Error finding container ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835: Status 404 returned error can't find the container with id ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835 Oct 30 22:31:52.613984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:52.613873 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7"] Oct 30 22:31:53.526347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:53.521783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" event={"ID":"2c546161-cc87-4a97-8101-532ce6ae58f9","Type":"ContainerStarted","Data":"8fd5e34df0ca770ff51b18fcfddd38a1474d477e401c6b271215ab60dbef0406"} Oct 30 22:31:53.526347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:53.521856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" event={"ID":"2c546161-cc87-4a97-8101-532ce6ae58f9","Type":"ContainerStarted","Data":"ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835"} Oct 30 22:31:53.614327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:53.613098 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:53.614327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:53.613156 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:53.621651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:53.616865 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" podStartSLOduration=2.616846033 podStartE2EDuration="2.616846033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:53.545208299 +0000 UTC m=+3137.495163245" watchObservedRunningTime="2026-10-30 22:31:53.616846033 +0000 UTC m=+3137.566800981" Oct 30 22:31:54.190891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:54.190848 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3b32e3-67c2-4f60-813b-aca0436d24b1" path="/var/lib/kubelet/pods/db3b32e3-67c2-4f60-813b-aca0436d24b1/volumes" Oct 30 22:31:54.532029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:54.531935 5036 generic.go:356] "Generic (PLEG): container finished" podID="2c546161-cc87-4a97-8101-532ce6ae58f9" containerID="8fd5e34df0ca770ff51b18fcfddd38a1474d477e401c6b271215ab60dbef0406" exitCode=0 Oct 30 22:31:54.532417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:54.532021 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" event={"ID":"2c546161-cc87-4a97-8101-532ce6ae58f9","Type":"ContainerDied","Data":"8fd5e34df0ca770ff51b18fcfddd38a1474d477e401c6b271215ab60dbef0406"} Oct 30 22:31:55.748530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.748501 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:55.845259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.845172 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\" (UniqueName: \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\") pod \"2c546161-cc87-4a97-8101-532ce6ae58f9\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " Oct 30 22:31:55.845462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.845409 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\") pod \"2c546161-cc87-4a97-8101-532ce6ae58f9\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " Oct 30 22:31:55.845770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.845747 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\") pod \"2c546161-cc87-4a97-8101-532ce6ae58f9\" (UID: \"2c546161-cc87-4a97-8101-532ce6ae58f9\") " Oct 30 22:31:55.847131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.847065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca" pod "2c546161-cc87-4a97-8101-532ce6ae58f9" (UID: "2c546161-cc87-4a97-8101-532ce6ae58f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:55.850385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.850252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca" pod "2c546161-cc87-4a97-8101-532ce6ae58f9" (UID: "2c546161-cc87-4a97-8101-532ce6ae58f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:55.851326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.851288 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk" pod "2c546161-cc87-4a97-8101-532ce6ae58f9" (UID: "2c546161-cc87-4a97-8101-532ce6ae58f9"). InnerVolumeSpecName "kube-api-access-w6hfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:55.864180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.864152 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:31:55.864464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.864423 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" containerID="cri-o://814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3" gracePeriod=30 Oct 30 22:31:55.994291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.991898 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:55.994291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.991938 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hfk\" (UniqueName: \"kubernetes.io/projected/2c546161-cc87-4a97-8101-532ce6ae58f9-kube-api-access-w6hfk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:55.994291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:55.991958 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c546161-cc87-4a97-8101-532ce6ae58f9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.048291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.048116 5036 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:56.054340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.048517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:56.428373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.424695 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn"] Oct 30 22:31:56.428373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.426240 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2c546161-cc87-4a97-8101-532ce6ae58f9" containerName="revoke-keys" Oct 30 22:31:56.428373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.426284 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c546161-cc87-4a97-8101-532ce6ae58f9" containerName="revoke-keys" Oct 30 22:31:56.428373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.426495 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c546161-cc87-4a97-8101-532ce6ae58f9" containerName="revoke-keys" Oct 30 22:31:56.434035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.434011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.437405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.437323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn"] Oct 30 22:31:56.515803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.515759 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.515996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.515977 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.516051 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.516020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnr5b\" (UniqueName: \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.585721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.585689 5036 generic.go:356] "Generic (PLEG): container finished" podID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerID="5a64d815843df9d12cf808d95b3cff3be503ce058bb8ac284af3c81a99d46475" exitCode=0 Oct 30 22:31:56.585834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.585794 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" event={"ID":"7e38752a-b9f0-453f-b884-f15c3522c7b2","Type":"ContainerDied","Data":"5a64d815843df9d12cf808d95b3cff3be503ce058bb8ac284af3c81a99d46475"} Oct 30 22:31:56.587633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.587607 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" event={"ID":"2c546161-cc87-4a97-8101-532ce6ae58f9","Type":"ContainerDied","Data":"ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835"} Oct 30 22:31:56.587753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.587654 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce72e56098d59a0f4281a745d3ff2111c5e797affb98b2ff3a29fdfb0cdf0835" Oct 30 22:31:56.587753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.587703 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7" Oct 30 22:31:56.591406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.591242 5036 generic.go:356] "Generic (PLEG): container finished" podID="43ee1dcb-902e-45fc-957f-c3069627d439" containerID="4685d4b9acb233f3312f6023502ac733a528c01ab7006ee14cfcdd7fdb31a9d6" exitCode=0 Oct 30 22:31:56.591544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.591518 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" event={"ID":"43ee1dcb-902e-45fc-957f-c3069627d439","Type":"ContainerDied","Data":"4685d4b9acb233f3312f6023502ac733a528c01ab7006ee14cfcdd7fdb31a9d6"} Oct 30 22:31:56.594356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.594331 5036 generic.go:356] "Generic (PLEG): container finished" podID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerID="2ec1b170c6fd2a13a2efe178f45543f2e0d5044c131a199d57634cdde3b2b417" exitCode=0 Oct 30 22:31:56.594599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.594577 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" event={"ID":"87b77dbe-fd9d-416d-a02f-66a8eaf39125","Type":"ContainerDied","Data":"2ec1b170c6fd2a13a2efe178f45543f2e0d5044c131a199d57634cdde3b2b417"} Oct 30 22:31:56.597822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.597802 5036 generic.go:356] "Generic (PLEG): container finished" podID="71ecec29-954d-4306-98fe-62d3e2085f81" containerID="cecdfb7226fca9d79144767e66f0812ea60298c335980a91e809478a4fb4792c" exitCode=0 Oct 30 22:31:56.597909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.597868 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerDied","Data":"cecdfb7226fca9d79144767e66f0812ea60298c335980a91e809478a4fb4792c"} Oct 30 22:31:56.597949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.597914 5036 scope.go:122] "RemoveContainer" containerID="977b69c03187d6938e9460990fc7d13b2bbdefa1d64d63622c7882324b8cc88b" Oct 30 22:31:56.600019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.600001 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:31:56.618139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.618111 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.618350 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.618330 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.618487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.618373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr5b\" (UniqueName: \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.619309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.619289 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.622391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.622369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.652368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.651669 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr5b\" (UniqueName: \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.719786 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.719868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.719982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720015 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720065 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720218 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.720346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720251 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.722088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720510 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.722088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.720564 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\") pod \"71ecec29-954d-4306-98fe-62d3e2085f81\" (UID: \"71ecec29-954d-4306-98fe-62d3e2085f81\") " Oct 30 22:31:56.723828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.723767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.725483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.725448 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.727087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.727056 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.727356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.727335 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.727876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.727842 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.730029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.729774 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.730029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.729788 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.737760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.737721 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.740257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.738784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z" pod "71ecec29-954d-4306-98fe-62d3e2085f81" (UID: "71ecec29-954d-4306-98fe-62d3e2085f81"). InnerVolumeSpecName "kube-api-access-2478z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.767788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.767761 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:31:56.790165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.788515 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:31:56.793470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.793445 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:31:56.824309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824226 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824290 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824311 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824324 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824339 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824354 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2478z\" (UniqueName: \"kubernetes.io/projected/71ecec29-954d-4306-98fe-62d3e2085f81-kube-api-access-2478z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824369 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ecec29-954d-4306-98fe-62d3e2085f81-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824383 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ecec29-954d-4306-98fe-62d3e2085f81-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.824649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.824395 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ecec29-954d-4306-98fe-62d3e2085f81-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.841432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.841326 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:31:56.925123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925089 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.925473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.925618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.925769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925753 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.926357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.926616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.925841 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.926616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.926201 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.927093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927046 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.927322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927300 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.927483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927466 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.927614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927600 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.927736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927723 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.927846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.927997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.927982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.928114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.928702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928662 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.928823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.928823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928711 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.928823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928750 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.928823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928792 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928894 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928925 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928959 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.928992 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929003 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929017 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929057 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929141 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929204 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929239 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\") pod \"7e38752a-b9f0-453f-b884-f15c3522c7b2\" (UID: \"7e38752a-b9f0-453f-b884-f15c3522c7b2\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929388 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929435 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\") pod \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\" (UID: \"87b77dbe-fd9d-416d-a02f-66a8eaf39125\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929473 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.929519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.929514 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\") pod \"43ee1dcb-902e-45fc-957f-c3069627d439\" (UID: \"43ee1dcb-902e-45fc-957f-c3069627d439\") " Oct 30 22:31:56.930426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930405 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930433 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930448 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e38752a-b9f0-453f-b884-f15c3522c7b2-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930463 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930478 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee1dcb-902e-45fc-957f-c3069627d439-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930493 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930508 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.930866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.930523 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.932756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.932370 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.932756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.932620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.932967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.932819 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.933031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.932977 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.933159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.933138 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.934105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.933605 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "kube-api-access-nhxlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.934105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.933862 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.936941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.936481 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.936941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.936690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.939094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.938878 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.939177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.939139 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "kube-api-access-4xhhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.941141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.941117 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.943440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943246 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.943440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943315 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.943440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.943696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.943696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4" pod "87b77dbe-fd9d-416d-a02f-66a8eaf39125" (UID: "87b77dbe-fd9d-416d-a02f-66a8eaf39125"). InnerVolumeSpecName "kube-api-access-ssqp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.943929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.943904 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token" pod "7e38752a-b9f0-453f-b884-f15c3522c7b2" (UID: "7e38752a-b9f0-453f-b884-f15c3522c7b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.946442 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:56.946354 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo" pod "43ee1dcb-902e-45fc-957f-c3069627d439" (UID: "43ee1dcb-902e-45fc-957f-c3069627d439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031855 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031884 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031899 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031908 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e38752a-b9f0-453f-b884-f15c3522c7b2-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031918 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssqp4\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-kube-api-access-ssqp4\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031927 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.031932 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031937 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031947 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031956 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031966 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031981 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee1dcb-902e-45fc-957f-c3069627d439-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031991 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.031999 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032008 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032016 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhhr\" (UniqueName: \"kubernetes.io/projected/7e38752a-b9f0-453f-b884-f15c3522c7b2-kube-api-access-4xhhr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032024 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b77dbe-fd9d-416d-a02f-66a8eaf39125-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032033 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b77dbe-fd9d-416d-a02f-66a8eaf39125-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032044 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee1dcb-902e-45fc-957f-c3069627d439-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.032456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.032057 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxlk\" (UniqueName: \"kubernetes.io/projected/43ee1dcb-902e-45fc-957f-c3069627d439-kube-api-access-nhxlk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.091728 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:31:57.091684 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0b9d04_c904_4f76_a94a_95140bb1694e.slice/crio-9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269 WatchSource:0}: Error finding container 9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269: Status 404 returned error can't find the container with id 9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269 Oct 30 22:31:57.092349 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.092319 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn"] Oct 30 22:31:57.608462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.608423 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" Oct 30 22:31:57.608663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.608459 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667" event={"ID":"43ee1dcb-902e-45fc-957f-c3069627d439","Type":"ContainerDied","Data":"09ca63e1889325f74fb6ffaf34a6481043d6d6c821f506a653e65c40f156d61f"} Oct 30 22:31:57.608663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.608522 5036 scope.go:122] "RemoveContainer" containerID="4685d4b9acb233f3312f6023502ac733a528c01ab7006ee14cfcdd7fdb31a9d6" Oct 30 22:31:57.612802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.610395 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" Oct 30 22:31:57.612802 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.610403 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw" event={"ID":"87b77dbe-fd9d-416d-a02f-66a8eaf39125","Type":"ContainerDied","Data":"232ed8bce16bcd545452c136cd60889b1807817f41e94c17dcac32f5cb5f55a6"} Oct 30 22:31:57.623752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.621892 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" event={"ID":"71ecec29-954d-4306-98fe-62d3e2085f81","Type":"ContainerDied","Data":"2ae2775dfee94a5d5f04092be475f3fb777b1bffd985ee3d1fe9c5d786e6f8e0"} Oct 30 22:31:57.623752 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.621988 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv" Oct 30 22:31:57.625490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.625467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" event={"ID":"cb0b9d04-c904-4f76-a94a-95140bb1694e","Type":"ContainerStarted","Data":"9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269"} Oct 30 22:31:57.628373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.628318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" event={"ID":"7e38752a-b9f0-453f-b884-f15c3522c7b2","Type":"ContainerDied","Data":"66acf984f982b0e230e5b5bbfd1f833adef82756587ed927828bce9275f83e99"} Oct 30 22:31:57.628373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.628362 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm" Oct 30 22:31:57.628859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.628841 5036 scope.go:122] "RemoveContainer" containerID="2ec1b170c6fd2a13a2efe178f45543f2e0d5044c131a199d57634cdde3b2b417" Oct 30 22:31:57.643375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.643352 5036 scope.go:122] "RemoveContainer" containerID="cecdfb7226fca9d79144767e66f0812ea60298c335980a91e809478a4fb4792c" Oct 30 22:31:57.655629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.655612 5036 scope.go:122] "RemoveContainer" containerID="5a64d815843df9d12cf808d95b3cff3be503ce058bb8ac284af3c81a99d46475" Oct 30 22:31:57.666880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.666846 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:31:57.673343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.673319 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-rrhnv"] Oct 30 22:31:57.680644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.680607 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:31:57.685402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.685378 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-c55nw"] Oct 30 22:31:57.695710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.695686 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:31:57.701831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.701774 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-2j667"] Oct 30 22:31:57.718021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.717995 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:31:57.718139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:57.718044 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-4p8qm"] Oct 30 22:31:58.192075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.192032 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" path="/var/lib/kubelet/pods/43ee1dcb-902e-45fc-957f-c3069627d439/volumes" Oct 30 22:31:58.193057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.193034 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" path="/var/lib/kubelet/pods/71ecec29-954d-4306-98fe-62d3e2085f81/volumes" Oct 30 22:31:58.193756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.193735 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" path="/var/lib/kubelet/pods/7e38752a-b9f0-453f-b884-f15c3522c7b2/volumes" Oct 30 22:31:58.194998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.194978 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" path="/var/lib/kubelet/pods/87b77dbe-fd9d-416d-a02f-66a8eaf39125/volumes" Oct 30 22:31:58.642748 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.642714 5036 generic.go:356] "Generic (PLEG): container finished" podID="cb0b9d04-c904-4f76-a94a-95140bb1694e" containerID="afb9ada8d3df89529ec4bacc37e21846a1f1bc0a40d1b53b5f5f397c4d5b875a" exitCode=0 Oct 30 22:31:58.642958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:58.642773 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" event={"ID":"cb0b9d04-c904-4f76-a94a-95140bb1694e","Type":"ContainerDied","Data":"afb9ada8d3df89529ec4bacc37e21846a1f1bc0a40d1b53b5f5f397c4d5b875a"} Oct 30 22:31:59.433843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.433810 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:31:59.434328 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.434049 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" containerID="cri-o://188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1" gracePeriod=30 Oct 30 22:31:59.928556 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.928521 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p"] Oct 30 22:31:59.929306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929280 5036 cpu_manager.go:414] "containerMap: removing container" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" Oct 30 22:31:59.929306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929301 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929337 5036 cpu_manager.go:414] "containerMap: removing container" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929342 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929357 5036 cpu_manager.go:414] "containerMap: removing container" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929363 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929372 5036 cpu_manager.go:414] "containerMap: removing container" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929377 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929408 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929413 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929516 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e38752a-b9f0-453f-b884-f15c3522c7b2" containerName="istio-proxy" Oct 30 22:31:59.929521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929526 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b77dbe-fd9d-416d-a02f-66a8eaf39125" containerName="istio-proxy" Oct 30 22:31:59.929882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929539 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.929882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929545 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ee1dcb-902e-45fc-957f-c3069627d439" containerName="istio-proxy" Oct 30 22:31:59.929882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.929551 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ecec29-954d-4306-98fe-62d3e2085f81" containerName="istio-proxy" Oct 30 22:31:59.949667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.949632 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p"] Oct 30 22:31:59.949835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:31:59.949633 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.018042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.018019 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:32:00.113546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.113506 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\" (UniqueName: \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\") pod \"cb0b9d04-c904-4f76-a94a-95140bb1694e\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " Oct 30 22:32:00.113746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.113634 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\") pod \"cb0b9d04-c904-4f76-a94a-95140bb1694e\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " Oct 30 22:32:00.113746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.113701 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\") pod \"cb0b9d04-c904-4f76-a94a-95140bb1694e\" (UID: \"cb0b9d04-c904-4f76-a94a-95140bb1694e\") " Oct 30 22:32:00.114001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.113972 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca" pod "cb0b9d04-c904-4f76-a94a-95140bb1694e" (UID: "cb0b9d04-c904-4f76-a94a-95140bb1694e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:00.114397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.114373 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.114578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.114562 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.114706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.114688 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbds\" (UniqueName: \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.114829 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.114811 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:00.116092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.116065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca" pod "cb0b9d04-c904-4f76-a94a-95140bb1694e" (UID: "cb0b9d04-c904-4f76-a94a-95140bb1694e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:00.116202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.116121 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b" pod "cb0b9d04-c904-4f76-a94a-95140bb1694e" (UID: "cb0b9d04-c904-4f76-a94a-95140bb1694e"). InnerVolumeSpecName "kube-api-access-bnr5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:00.215846 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.215813 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbds\" (UniqueName: \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.216032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.216019 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.216143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.216131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.216314 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.216293 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnr5b\" (UniqueName: \"kubernetes.io/projected/cb0b9d04-c904-4f76-a94a-95140bb1694e-kube-api-access-bnr5b\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:00.216355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.216326 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b9d04-c904-4f76-a94a-95140bb1694e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:00.216792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.216766 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.218737 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.218712 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.228086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.228019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbds\" (UniqueName: \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.316515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.316483 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:00.526936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.526839 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:00.613433 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:00.613401 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd385a21c_b4c0_4518_9149_406629986a17.slice/crio-60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57 WatchSource:0}: Error finding container 60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57: Status 404 returned error can't find the container with id 60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57 Oct 30 22:32:00.614308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.614283 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p"] Oct 30 22:32:00.725313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.723855 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" Oct 30 22:32:00.725313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.723882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn" event={"ID":"cb0b9d04-c904-4f76-a94a-95140bb1694e","Type":"ContainerDied","Data":"9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269"} Oct 30 22:32:00.725313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.724072 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9118eac5e88a740c216ccff18818cfee008a916d80af179d07ca5a6cec785269" Oct 30 22:32:00.725962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:00.725926 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" event={"ID":"d385a21c-b4c0-4518-9149-406629986a17","Type":"ContainerStarted","Data":"60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57"} Oct 30 22:32:01.174330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.174302 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:32:01.233511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233475 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.233723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233579 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.233723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233661 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.233723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233716 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.233921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.233921 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.233864 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.234033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234000 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.234094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.234194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234151 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\") pod \"3e0231bc-fcb6-4ab5-80a5-498724efea61\" (UID: \"3e0231bc-fcb6-4ab5-80a5-498724efea61\") " Oct 30 22:32:01.234994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234149 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:01.234994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:01.234994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.234955 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:01.235551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.235514 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:01.236054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.236031 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:01.238793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.238753 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:01.239297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.239238 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:01.239526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.239506 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:01.239602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.239551 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944" pod "3e0231bc-fcb6-4ab5-80a5-498724efea61" (UID: "3e0231bc-fcb6-4ab5-80a5-498724efea61"). InnerVolumeSpecName "kube-api-access-px944". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:01.336455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336410 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336455 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336472 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336485 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336494 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336503 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0231bc-fcb6-4ab5-80a5-498724efea61-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336512 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px944\" (UniqueName: \"kubernetes.io/projected/3e0231bc-fcb6-4ab5-80a5-498724efea61-kube-api-access-px944\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336522 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.336672 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.336531 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0231bc-fcb6-4ab5-80a5-498724efea61-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:01.739114 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.739072 5036 generic.go:356] "Generic (PLEG): container finished" podID="d385a21c-b4c0-4518-9149-406629986a17" containerID="dad53b07334ed443eaed392f44922b496bbdbac3de31834c092bff49b528673f" exitCode=0 Oct 30 22:32:01.739596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.739162 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" event={"ID":"d385a21c-b4c0-4518-9149-406629986a17","Type":"ContainerDied","Data":"dad53b07334ed443eaed392f44922b496bbdbac3de31834c092bff49b528673f"} Oct 30 22:32:01.741318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.741295 5036 generic.go:356] "Generic (PLEG): container finished" podID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerID="814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3" exitCode=0 Oct 30 22:32:01.741445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.741373 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" Oct 30 22:32:01.741445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.741396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerDied","Data":"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3"} Oct 30 22:32:01.741541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.741449 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l" event={"ID":"3e0231bc-fcb6-4ab5-80a5-498724efea61","Type":"ContainerDied","Data":"1aefa4149e7208e256a51362b74741d859e8323c09d6634351f70f450457f4f6"} Oct 30 22:32:01.741541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.741473 5036 scope.go:122] "RemoveContainer" containerID="814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3" Oct 30 22:32:01.760421 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.760397 5036 scope.go:122] "RemoveContainer" containerID="2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7" Oct 30 22:32:01.780356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.780329 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:32:01.781961 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.781940 5036 scope.go:122] "RemoveContainer" containerID="814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3" Oct 30 22:32:01.782337 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:01.782297 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3\": container with ID starting with 814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3 not found: ID does not exist" containerID="814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3" Oct 30 22:32:01.782457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.782342 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3"} err="failed to get container status \"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3\": rpc error: code = NotFound desc = could not find container \"814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3\": container with ID starting with 814429b0ab43e5324df4935c07f2296880e50c2dfc60d718e52b7bd440ef12e3 not found: ID does not exist" Oct 30 22:32:01.782457 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.782362 5036 scope.go:122] "RemoveContainer" containerID="2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7" Oct 30 22:32:01.782628 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:01.782608 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7\": container with ID starting with 2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7 not found: ID does not exist" containerID="2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7" Oct 30 22:32:01.782696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.782638 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7"} err="failed to get container status \"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7\": rpc error: code = NotFound desc = could not find container \"2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7\": container with ID starting with 2433541b1f0467316fce81348284ace55399a18e693d442e40289ee827d45cf7 not found: ID does not exist" Oct 30 22:32:01.785535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:01.785510 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-58d5fa-openshift-default-76b76bf447-t5b2l"] Oct 30 22:32:02.321341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.321214 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" path="/var/lib/kubelet/pods/3e0231bc-fcb6-4ab5-80a5-498724efea61/volumes" Oct 30 22:32:02.496134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.496110 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:32:02.563562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.563521 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\" (UniqueName: \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\") pod \"5454cfbd-cd38-401e-a469-7655539d3e53\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " Oct 30 22:32:02.563771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.563666 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\") pod \"5454cfbd-cd38-401e-a469-7655539d3e53\" (UID: \"5454cfbd-cd38-401e-a469-7655539d3e53\") " Oct 30 22:32:02.565056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.565019 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume" pod "5454cfbd-cd38-401e-a469-7655539d3e53" (UID: "5454cfbd-cd38-401e-a469-7655539d3e53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:02.572857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.572782 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2" pod "5454cfbd-cd38-401e-a469-7655539d3e53" (UID: "5454cfbd-cd38-401e-a469-7655539d3e53"). InnerVolumeSpecName "kube-api-access-jf8d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:02.665904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.665872 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5454cfbd-cd38-401e-a469-7655539d3e53-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:02.665904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.665903 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8d2\" (UniqueName: \"kubernetes.io/projected/5454cfbd-cd38-401e-a469-7655539d3e53-kube-api-access-jf8d2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:02.754680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.754643 5036 generic.go:356] "Generic (PLEG): container finished" podID="5454cfbd-cd38-401e-a469-7655539d3e53" containerID="636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190" exitCode=0 Oct 30 22:32:02.755154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.754718 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" Oct 30 22:32:02.755154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.754732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" event={"ID":"5454cfbd-cd38-401e-a469-7655539d3e53","Type":"ContainerDied","Data":"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190"} Oct 30 22:32:02.755154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.754817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc" event={"ID":"5454cfbd-cd38-401e-a469-7655539d3e53","Type":"ContainerDied","Data":"a28f2cb788bd5323029df1681f6eb2ebe45c5aaa91e3660e468c65454499087d"} Oct 30 22:32:02.755154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.754840 5036 scope.go:122] "RemoveContainer" containerID="636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190" Oct 30 22:32:02.773043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.773020 5036 scope.go:122] "RemoveContainer" containerID="636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190" Oct 30 22:32:02.773426 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:02.773401 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190\": container with ID starting with 636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190 not found: ID does not exist" containerID="636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190" Oct 30 22:32:02.773539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.773433 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190"} err="failed to get container status \"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190\": rpc error: code = NotFound desc = could not find container \"636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190\": container with ID starting with 636a2661f1196019f49a2f50104b9e6cc8c0b01dacdb85390b0af6a351a54190 not found: ID does not exist" Oct 30 22:32:02.785656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.785630 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:32:02.793049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.793023 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-120cb1-868dd48479-twqkc"] Oct 30 22:32:02.925837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.925809 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:02.934404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.934374 5036 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:02.934548 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.934424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:02.972330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.970443 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\" (UniqueName: \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\") pod \"d385a21c-b4c0-4518-9149-406629986a17\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " Oct 30 22:32:02.972330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.970509 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\") pod \"d385a21c-b4c0-4518-9149-406629986a17\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " Oct 30 22:32:02.972330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.970744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\") pod \"d385a21c-b4c0-4518-9149-406629986a17\" (UID: \"d385a21c-b4c0-4518-9149-406629986a17\") " Oct 30 22:32:02.984636 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.978194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca" pod "d385a21c-b4c0-4518-9149-406629986a17" (UID: "d385a21c-b4c0-4518-9149-406629986a17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:02.998878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.998038 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca" pod "d385a21c-b4c0-4518-9149-406629986a17" (UID: "d385a21c-b4c0-4518-9149-406629986a17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:02.998878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:02.998696 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds" pod "d385a21c-b4c0-4518-9149-406629986a17" (UID: "d385a21c-b4c0-4518-9149-406629986a17"). InnerVolumeSpecName "kube-api-access-5tbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:03.072554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.072523 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d385a21c-b4c0-4518-9149-406629986a17-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:03.072554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.072553 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbds\" (UniqueName: \"kubernetes.io/projected/d385a21c-b4c0-4518-9149-406629986a17-kube-api-access-5tbds\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:03.072749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.072572 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d385a21c-b4c0-4518-9149-406629986a17-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:03.797495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.797465 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" Oct 30 22:32:03.797495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.797495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p" event={"ID":"d385a21c-b4c0-4518-9149-406629986a17","Type":"ContainerDied","Data":"60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57"} Oct 30 22:32:03.798028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:03.797552 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b01f86ca563a92b2d8f4e84dc77223306d562c3eb272dab362c6d1a58e4b57" Oct 30 22:32:04.192080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.192038 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" path="/var/lib/kubelet/pods/5454cfbd-cd38-401e-a469-7655539d3e53/volumes" Oct 30 22:32:04.728313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.728285 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:32:04.798199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798164 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798235 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798327 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798404 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798455 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798498 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.798750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.799243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.798776 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\") pod \"9d40a53b-e3fe-49dd-a0a0-9720de857402\" (UID: \"9d40a53b-e3fe-49dd-a0a0-9720de857402\") " Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.799751 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.799953 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.800126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.800458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.800520 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801078 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801310 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801398 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801420 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801436 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801451 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801464 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d40a53b-e3fe-49dd-a0a0-9720de857402-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801476 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.801646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.801556 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "kube-api-access-xj7bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:04.805445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.805407 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy" pod "9d40a53b-e3fe-49dd-a0a0-9720de857402" (UID: "9d40a53b-e3fe-49dd-a0a0-9720de857402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:04.817521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.817066 5036 generic.go:356] "Generic (PLEG): container finished" podID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerID="188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1" exitCode=0 Oct 30 22:32:04.817521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.817123 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" Oct 30 22:32:04.817521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.817360 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerDied","Data":"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1"} Oct 30 22:32:04.817521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.817403 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq" event={"ID":"9d40a53b-e3fe-49dd-a0a0-9720de857402","Type":"ContainerDied","Data":"0891736f5ba6b458d0aa47463505a82107d4c149d792bcb8fac1962ba53c9694"} Oct 30 22:32:04.817521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.817435 5036 scope.go:122] "RemoveContainer" containerID="188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1" Oct 30 22:32:04.848511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.848490 5036 scope.go:122] "RemoveContainer" containerID="d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6" Oct 30 22:32:04.860305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.860241 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:32:04.863422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.863401 5036 scope.go:122] "RemoveContainer" containerID="188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1" Oct 30 22:32:04.863756 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:04.863735 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1\": container with ID starting with 188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1 not found: ID does not exist" containerID="188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1" Oct 30 22:32:04.863839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.863771 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1"} err="failed to get container status \"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1\": rpc error: code = NotFound desc = could not find container \"188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1\": container with ID starting with 188dd74cc5a9900f15a0b823644e8e8db16ad1f63e7548a7be27859af3e2fbd1 not found: ID does not exist" Oct 30 22:32:04.863839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.863797 5036 scope.go:122] "RemoveContainer" containerID="d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6" Oct 30 22:32:04.864150 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:04.864127 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6\": container with ID starting with d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6 not found: ID does not exist" containerID="d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6" Oct 30 22:32:04.864198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.864160 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6"} err="failed to get container status \"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6\": rpc error: code = NotFound desc = could not find container \"d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6\": container with ID starting with d71881015c51fc615b5ff8477f46c790db88e5a3ed5ab034b9b647f2833b14a6 not found: ID does not exist" Oct 30 22:32:04.866086 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.866069 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-g6xpq"] Oct 30 22:32:04.902935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.902911 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj7bn\" (UniqueName: \"kubernetes.io/projected/9d40a53b-e3fe-49dd-a0a0-9720de857402-kube-api-access-xj7bn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.902935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.902935 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:04.903056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:04.902947 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d40a53b-e3fe-49dd-a0a0-9720de857402-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:06.190194 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:06.190159 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" path="/var/lib/kubelet/pods/9d40a53b-e3fe-49dd-a0a0-9720de857402/volumes" Oct 30 22:32:07.355289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:07.353397 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7"] Oct 30 22:32:07.375803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:07.364318 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b2bb0-sh4j7"] Oct 30 22:32:08.189853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:08.189820 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c546161-cc87-4a97-8101-532ce6ae58f9" path="/var/lib/kubelet/pods/2c546161-cc87-4a97-8101-532ce6ae58f9/volumes" Oct 30 22:32:12.908291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:12.907752 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p"] Oct 30 22:32:12.915541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:12.915305 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5bdf-xzt6p"] Oct 30 22:32:14.190671 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:14.190638 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d385a21c-b4c0-4518-9149-406629986a17" path="/var/lib/kubelet/pods/d385a21c-b4c0-4518-9149-406629986a17/volumes" Oct 30 22:32:14.902929 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:14.902890 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:32:14.903484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:14.903441 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" containerID="cri-o://980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608" gracePeriod=30 Oct 30 22:32:15.377398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.377322 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c"] Oct 30 22:32:15.378117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378102 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:15.378164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378119 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:15.378164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378132 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:15.378164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378137 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:15.378164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378154 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d385a21c-b4c0-4518-9149-406629986a17" containerName="revoke-keys" Oct 30 22:32:15.378164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378160 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d385a21c-b4c0-4518-9149-406629986a17" containerName="revoke-keys" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378173 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cb0b9d04-c904-4f76-a94a-95140bb1694e" containerName="revoke-keys" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378178 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0b9d04-c904-4f76-a94a-95140bb1694e" containerName="revoke-keys" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378202 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378208 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378217 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378221 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378346 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:15.378359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378354 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5454cfbd-cd38-401e-a469-7655539d3e53" containerName="payload-pre-processing" Oct 30 22:32:15.378607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378366 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:15.378607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378374 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d385a21c-b4c0-4518-9149-406629986a17" containerName="revoke-keys" Oct 30 22:32:15.378607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378382 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0b9d04-c904-4f76-a94a-95140bb1694e" containerName="revoke-keys" Oct 30 22:32:15.378607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.378391 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:15.383379 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.383360 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.387320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.387278 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:15.387464 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.387319 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:15.390728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.390707 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c"] Oct 30 22:32:15.437293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.418507 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:32:15.437293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.418747 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:15.449678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.449630 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.450008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.449983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.450534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.450505 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64tn\" (UniqueName: \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.552256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.552222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64tn\" (UniqueName: \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.552447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.552332 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.552447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.552377 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.553221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.553191 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.554996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.554975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.564142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.564117 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64tn\" (UniqueName: \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:15.760282 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:15.760235 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:16.054947 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:16.054914 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a86b155_c470_46f3_8991_dee0b613ba89.slice/crio-d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae WatchSource:0}: Error finding container d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae: Status 404 returned error can't find the container with id d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae Oct 30 22:32:16.058586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:16.056608 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c"] Oct 30 22:32:16.102461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:16.102433 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" event={"ID":"7a86b155-c470-46f3-8991-dee0b613ba89","Type":"ContainerStarted","Data":"d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae"} Oct 30 22:32:17.167290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:17.162219 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" event={"ID":"7a86b155-c470-46f3-8991-dee0b613ba89","Type":"ContainerStarted","Data":"8a4d09939068149201d305ea5c73df5e1e81ce3e1df54dbcafe955590df8c9fd"} Oct 30 22:32:17.190535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:17.190458 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn"] Oct 30 22:32:17.195977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:17.195947 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bacea0-8s2nn"] Oct 30 22:32:17.198481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:17.198436 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" podStartSLOduration=2.198422533 podStartE2EDuration="2.198422533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:17.188027055 +0000 UTC m=+3161.137982003" watchObservedRunningTime="2026-10-30 22:32:17.198422533 +0000 UTC m=+3161.148377501" Oct 30 22:32:18.104233 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:18.104192 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:32:18.173537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:18.173495 5036 generic.go:356] "Generic (PLEG): container finished" podID="7a86b155-c470-46f3-8991-dee0b613ba89" containerID="8a4d09939068149201d305ea5c73df5e1e81ce3e1df54dbcafe955590df8c9fd" exitCode=0 Oct 30 22:32:18.173937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:18.173584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" event={"ID":"7a86b155-c470-46f3-8991-dee0b613ba89","Type":"ContainerDied","Data":"8a4d09939068149201d305ea5c73df5e1e81ce3e1df54dbcafe955590df8c9fd"} Oct 30 22:32:18.189673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:18.189650 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0b9d04-c904-4f76-a94a-95140bb1694e" path="/var/lib/kubelet/pods/cb0b9d04-c904-4f76-a94a-95140bb1694e/volumes" Oct 30 22:32:18.604851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:18.604816 5036 patch_prober.go:29] interesting pod/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:18.605008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:18.604880 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:19.376659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.376635 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:19.455397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.455301 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\") pod \"7a86b155-c470-46f3-8991-dee0b613ba89\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " Oct 30 22:32:19.455397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.455401 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\") pod \"7a86b155-c470-46f3-8991-dee0b613ba89\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " Oct 30 22:32:19.455664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.455511 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\" (UniqueName: \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\") pod \"7a86b155-c470-46f3-8991-dee0b613ba89\" (UID: \"7a86b155-c470-46f3-8991-dee0b613ba89\") " Oct 30 22:32:19.456238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.456179 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca" pod "7a86b155-c470-46f3-8991-dee0b613ba89" (UID: "7a86b155-c470-46f3-8991-dee0b613ba89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:19.457217 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.457189 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:19.459778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.459748 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn" pod "7a86b155-c470-46f3-8991-dee0b613ba89" (UID: "7a86b155-c470-46f3-8991-dee0b613ba89"). InnerVolumeSpecName "kube-api-access-d64tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:19.459894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.459857 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca" pod "7a86b155-c470-46f3-8991-dee0b613ba89" (UID: "7a86b155-c470-46f3-8991-dee0b613ba89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:19.577296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.576888 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a86b155-c470-46f3-8991-dee0b613ba89-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:19.577296 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:19.576929 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64tn\" (UniqueName: \"kubernetes.io/projected/7a86b155-c470-46f3-8991-dee0b613ba89-kube-api-access-d64tn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.174207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.174142 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:32:20.193814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.193783 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" Oct 30 22:32:20.193814 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.193786 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c" event={"ID":"7a86b155-c470-46f3-8991-dee0b613ba89","Type":"ContainerDied","Data":"d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae"} Oct 30 22:32:20.194004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.193847 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8ea8c93890cb4d2fc46ef5abfcd1e5dbd7f9602b853b7437b7ebe865920bdae" Oct 30 22:32:20.195974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.195950 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerID="980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608" exitCode=0 Oct 30 22:32:20.196107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.196003 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" Oct 30 22:32:20.196107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.196037 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerDied","Data":"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608"} Oct 30 22:32:20.196215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.196104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq" event={"ID":"2fe38da0-0f36-4681-8c00-eef3217bab9c","Type":"ContainerDied","Data":"11ea1109bf2f53619be420b0c491405e665594d98999b77a3dea4d944c0c46e2"} Oct 30 22:32:20.196215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.196138 5036 scope.go:122] "RemoveContainer" containerID="980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608" Oct 30 22:32:20.217246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.217226 5036 scope.go:122] "RemoveContainer" containerID="0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e" Oct 30 22:32:20.287331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287401 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287446 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287496 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287530 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287579 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.287730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287717 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:20.287954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.288021 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.287972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\") pod \"2fe38da0-0f36-4681-8c00-eef3217bab9c\" (UID: \"2fe38da0-0f36-4681-8c00-eef3217bab9c\") " Oct 30 22:32:20.288080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288015 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288284 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288716 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288931 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288955 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288973 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.288989 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.288311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.289002 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe38da0-0f36-4681-8c00-eef3217bab9c-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.289902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.289697 5036 scope.go:122] "RemoveContainer" containerID="980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608" Oct 30 22:32:20.290137 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:20.290115 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608\": container with ID starting with 980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608 not found: ID does not exist" containerID="980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608" Oct 30 22:32:20.290219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.290146 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608"} err="failed to get container status \"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608\": rpc error: code = NotFound desc = could not find container \"980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608\": container with ID starting with 980e579c04fdbd1283a6d83bdbf3d4a49d911f68fcce2928394f8bead7767608 not found: ID does not exist" Oct 30 22:32:20.290219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.290167 5036 scope.go:122] "RemoveContainer" containerID="0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e" Oct 30 22:32:20.290566 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:20.290543 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e\": container with ID starting with 0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e not found: ID does not exist" containerID="0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e" Oct 30 22:32:20.290660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.290574 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e"} err="failed to get container status \"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e\": rpc error: code = NotFound desc = could not find container \"0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e\": container with ID starting with 0527d8dc3380bc59e4c1c917e8573b2938817c5b8cab5dc89f3b894236d46a3e not found: ID does not exist" Oct 30 22:32:20.292158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.292126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:20.292292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.292204 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:20.292401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.292383 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:20.292831 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.292811 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz" pod "2fe38da0-0f36-4681-8c00-eef3217bab9c" (UID: "2fe38da0-0f36-4681-8c00-eef3217bab9c"). InnerVolumeSpecName "kube-api-access-w8phz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:20.390096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.390066 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.390096 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.390096 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8phz\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-kube-api-access-w8phz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.390541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.390110 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.390541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.390125 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe38da0-0f36-4681-8c00-eef3217bab9c-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:20.525837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.525783 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:32:20.532919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.532889 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4b2bb0-openshift-default-b4cdcbb57-fhklq"] Oct 30 22:32:20.692164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.692131 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:32:20.692431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:20.692392 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" containerID="cri-o://d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd" gracePeriod=30 Oct 30 22:32:21.188023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.187994 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4"] Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188809 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188826 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188851 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7a86b155-c470-46f3-8991-dee0b613ba89" containerName="revoke-keys" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188857 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a86b155-c470-46f3-8991-dee0b613ba89" containerName="revoke-keys" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188878 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188883 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d40a53b-e3fe-49dd-a0a0-9720de857402" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188892 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.188896 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.189026 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0231bc-fcb6-4ab5-80a5-498724efea61" containerName="istio-proxy" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.189037 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a86b155-c470-46f3-8991-dee0b613ba89" containerName="revoke-keys" Oct 30 22:32:21.189540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.189047 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:21.194523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.194504 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.198146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.197930 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:21.198146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.198005 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:21.209682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.205897 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4"] Oct 30 22:32:21.288564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.288532 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:21.288718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.288648 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:32:21.302958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.302920 5036 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:21.303097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.302976 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:21.329247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.329203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.329828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.329796 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxkf\" (UniqueName: \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.330126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.330090 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.432202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.432165 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.432628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.432240 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.432628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.432457 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxkf\" (UniqueName: \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.432817 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.432786 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.434890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.434864 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.446435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.446376 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxkf\" (UniqueName: \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\") pod \"maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.541372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.541335 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:21.864818 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:21.864776 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff0f379_1563_4ad5_be5c_a6d3db702726.slice/crio-8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a WatchSource:0}: Error finding container 8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a: Status 404 returned error can't find the container with id 8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a Oct 30 22:32:21.865240 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:21.865213 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4"] Oct 30 22:32:22.192057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:22.192023 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" path="/var/lib/kubelet/pods/2fe38da0-0f36-4681-8c00-eef3217bab9c/volumes" Oct 30 22:32:22.256963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:22.256934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" event={"ID":"bff0f379-1563-4ad5-be5c-a6d3db702726","Type":"ContainerStarted","Data":"d186f91e66cefaa9948e36d500d8d5cc538e3e6fd7f201372ca18a5dbb6a0338"} Oct 30 22:32:22.257116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:22.256981 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" event={"ID":"bff0f379-1563-4ad5-be5c-a6d3db702726","Type":"ContainerStarted","Data":"8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a"} Oct 30 22:32:22.278298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:22.278047 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" podStartSLOduration=1.278034586 podStartE2EDuration="1.278034586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:22.277052249 +0000 UTC m=+3166.227007196" watchObservedRunningTime="2026-10-30 22:32:22.278034586 +0000 UTC m=+3166.227989531" Oct 30 22:32:23.267994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:23.267959 5036 generic.go:356] "Generic (PLEG): container finished" podID="bff0f379-1563-4ad5-be5c-a6d3db702726" containerID="d186f91e66cefaa9948e36d500d8d5cc538e3e6fd7f201372ca18a5dbb6a0338" exitCode=0 Oct 30 22:32:23.268391 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:23.268000 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" event={"ID":"bff0f379-1563-4ad5-be5c-a6d3db702726","Type":"ContainerDied","Data":"d186f91e66cefaa9948e36d500d8d5cc538e3e6fd7f201372ca18a5dbb6a0338"} Oct 30 22:32:24.546451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.546428 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:24.595638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.595563 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9"] Oct 30 22:32:24.596825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.596801 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bff0f379-1563-4ad5-be5c-a6d3db702726" containerName="revoke-keys" Oct 30 22:32:24.596874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.596831 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff0f379-1563-4ad5-be5c-a6d3db702726" containerName="revoke-keys" Oct 30 22:32:24.597016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.597003 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe38da0-0f36-4681-8c00-eef3217bab9c" containerName="istio-proxy" Oct 30 22:32:24.597068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.597027 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff0f379-1563-4ad5-be5c-a6d3db702726" containerName="revoke-keys" Oct 30 22:32:24.601163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.601145 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.612496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.612451 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9"] Oct 30 22:32:24.635557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.635526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\" (UniqueName: \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\") pod \"bff0f379-1563-4ad5-be5c-a6d3db702726\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " Oct 30 22:32:24.635774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.635756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\") pod \"bff0f379-1563-4ad5-be5c-a6d3db702726\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " Oct 30 22:32:24.635859 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.635842 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\") pod \"bff0f379-1563-4ad5-be5c-a6d3db702726\" (UID: \"bff0f379-1563-4ad5-be5c-a6d3db702726\") " Oct 30 22:32:24.636169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.636137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca" pod "bff0f379-1563-4ad5-be5c-a6d3db702726" (UID: "bff0f379-1563-4ad5-be5c-a6d3db702726"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:24.636542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.636520 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.638667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.638639 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf" pod "bff0f379-1563-4ad5-be5c-a6d3db702726" (UID: "bff0f379-1563-4ad5-be5c-a6d3db702726"). InnerVolumeSpecName "kube-api-access-7rxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:24.640510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.640482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca" pod "bff0f379-1563-4ad5-be5c-a6d3db702726" (UID: "bff0f379-1563-4ad5-be5c-a6d3db702726"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:24.737383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.737341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l92z\" (UniqueName: \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.737557 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.737416 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.737678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.737661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.737777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.737766 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff0f379-1563-4ad5-be5c-a6d3db702726-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.737816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.737782 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxkf\" (UniqueName: \"kubernetes.io/projected/bff0f379-1563-4ad5-be5c-a6d3db702726-kube-api-access-7rxkf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.838688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.838649 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.838841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.838758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l92z\" (UniqueName: \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.838841 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.838815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.839637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.839540 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.841511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.841478 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.850585 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.850529 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l92z\" (UniqueName: \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\") pod \"maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:24.936383 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:24.936342 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:25.265506 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.265463 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9"] Oct 30 22:32:25.266031 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:25.266000 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f78f34_0320_4913_8e1f_81c48e74e9c4.slice/crio-cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d WatchSource:0}: Error finding container cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d: Status 404 returned error can't find the container with id cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d Oct 30 22:32:25.308188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.308158 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" event={"ID":"bff0f379-1563-4ad5-be5c-a6d3db702726","Type":"ContainerDied","Data":"8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a"} Oct 30 22:32:25.308378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.308219 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c75f13892c1c934edea334fe32ee65eceaf34cf4782dbb8c0b8a7a97ea7211a" Oct 30 22:32:25.308378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.308164 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4" Oct 30 22:32:25.312359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.312331 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" event={"ID":"03f78f34-0320-4913-8e1f-81c48e74e9c4","Type":"ContainerStarted","Data":"cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d"} Oct 30 22:32:25.516759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.516679 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:32:25.537698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:25.537660 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:32:26.191112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.191077 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:32:26.294994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.294963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295098 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295201 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295306 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295350 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295409 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:26.295567 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295463 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:26.295627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295628 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:26.295682 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295657 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\") pod \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\" (UID: \"fbbdf196-4f1a-4542-80d9-b83dafc230b1\") " Oct 30 22:32:26.295782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295726 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:26.295914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.295888 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:26.296610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.296584 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.296746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.296615 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.296746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.296632 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.296746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.296648 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.296746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.296687 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.297899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.297863 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:26.298012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.297949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:26.298237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.298212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:26.298597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.298574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z" pod "fbbdf196-4f1a-4542-80d9-b83dafc230b1" (UID: "fbbdf196-4f1a-4542-80d9-b83dafc230b1"). InnerVolumeSpecName "kube-api-access-wrv5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:26.324646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.324611 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" event={"ID":"03f78f34-0320-4913-8e1f-81c48e74e9c4","Type":"ContainerStarted","Data":"0935fd2b04f450a1d0333dceba7c6faeb4adebca6c52f0e9841debde5ddfa3aa"} Oct 30 22:32:26.326875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.326728 5036 generic.go:356] "Generic (PLEG): container finished" podID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerID="d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd" exitCode=0 Oct 30 22:32:26.326875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.326776 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" Oct 30 22:32:26.326875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.326834 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerDied","Data":"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd"} Oct 30 22:32:26.327192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.326893 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p" event={"ID":"fbbdf196-4f1a-4542-80d9-b83dafc230b1","Type":"ContainerDied","Data":"fc8c72ba52da2652135306506d9f5991a66d7ddd7c829d0df1da026701962a93"} Oct 30 22:32:26.327192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.326927 5036 scope.go:122] "RemoveContainer" containerID="d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd" Oct 30 22:32:26.340531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.340508 5036 scope.go:122] "RemoveContainer" containerID="3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404" Oct 30 22:32:26.345601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.345556 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" podStartSLOduration=2.345543593 podStartE2EDuration="2.345543593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:26.341300793 +0000 UTC m=+3170.291255769" watchObservedRunningTime="2026-10-30 22:32:26.345543593 +0000 UTC m=+3170.295498565" Oct 30 22:32:26.354313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.354296 5036 scope.go:122] "RemoveContainer" containerID="d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd" Oct 30 22:32:26.354625 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:26.354602 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd\": container with ID starting with d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd not found: ID does not exist" containerID="d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd" Oct 30 22:32:26.354697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.354632 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd"} err="failed to get container status \"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd\": rpc error: code = NotFound desc = could not find container \"d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd\": container with ID starting with d3b95e55d1c9210757f6a8f486e7356f43aa86dac84a001f678464b4647dd8fd not found: ID does not exist" Oct 30 22:32:26.354697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.354651 5036 scope.go:122] "RemoveContainer" containerID="3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404" Oct 30 22:32:26.355013 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:26.354988 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404\": container with ID starting with 3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404 not found: ID does not exist" containerID="3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404" Oct 30 22:32:26.355092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.355026 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404"} err="failed to get container status \"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404\": rpc error: code = NotFound desc = could not find container \"3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404\": container with ID starting with 3e7a557507c7ce5d105319a7409aece0b6e348f7e99018020dd2858ec9128404 not found: ID does not exist" Oct 30 22:32:26.359390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.359366 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:32:26.364111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.364088 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5bdf-openshift-default-8bbd74585-stl9p"] Oct 30 22:32:26.397996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.397937 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.397996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.397957 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.397996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.397967 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbbdf196-4f1a-4542-80d9-b83dafc230b1-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:26.397996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:26.397979 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrv5z\" (UniqueName: \"kubernetes.io/projected/fbbdf196-4f1a-4542-80d9-b83dafc230b1-kube-api-access-wrv5z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:27.351996 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:27.351955 5036 generic.go:356] "Generic (PLEG): container finished" podID="03f78f34-0320-4913-8e1f-81c48e74e9c4" containerID="0935fd2b04f450a1d0333dceba7c6faeb4adebca6c52f0e9841debde5ddfa3aa" exitCode=0 Oct 30 22:32:27.352503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:27.352052 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" event={"ID":"03f78f34-0320-4913-8e1f-81c48e74e9c4","Type":"ContainerDied","Data":"0935fd2b04f450a1d0333dceba7c6faeb4adebca6c52f0e9841debde5ddfa3aa"} Oct 30 22:32:28.204482 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.204446 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" path="/var/lib/kubelet/pods/fbbdf196-4f1a-4542-80d9-b83dafc230b1/volumes" Oct 30 22:32:28.539801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.539770 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:28.661293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.658179 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\") pod \"03f78f34-0320-4913-8e1f-81c48e74e9c4\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " Oct 30 22:32:28.661293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.658294 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\" (UniqueName: \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\") pod \"03f78f34-0320-4913-8e1f-81c48e74e9c4\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " Oct 30 22:32:28.661293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.658382 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\") pod \"03f78f34-0320-4913-8e1f-81c48e74e9c4\" (UID: \"03f78f34-0320-4913-8e1f-81c48e74e9c4\") " Oct 30 22:32:28.666079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.665047 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca" pod "03f78f34-0320-4913-8e1f-81c48e74e9c4" (UID: "03f78f34-0320-4913-8e1f-81c48e74e9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:28.666079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.665424 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca" pod "03f78f34-0320-4913-8e1f-81c48e74e9c4" (UID: "03f78f34-0320-4913-8e1f-81c48e74e9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:28.671289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.671228 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z" pod "03f78f34-0320-4913-8e1f-81c48e74e9c4" (UID: "03f78f34-0320-4913-8e1f-81c48e74e9c4"). InnerVolumeSpecName "kube-api-access-9l92z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:28.760480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.760384 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l92z\" (UniqueName: \"kubernetes.io/projected/03f78f34-0320-4913-8e1f-81c48e74e9c4-kube-api-access-9l92z\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.760480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.760418 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:28.760480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:28.760432 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f78f34-0320-4913-8e1f-81c48e74e9c4-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:29.373469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:29.373438 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" Oct 30 22:32:29.373646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:29.373467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9" event={"ID":"03f78f34-0320-4913-8e1f-81c48e74e9c4","Type":"ContainerDied","Data":"cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d"} Oct 30 22:32:29.373646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:29.373524 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf81bf50d1b4e1c018315092d314b59181ee86a5c73164910eddc454910a534d" Oct 30 22:32:30.517541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:30.517478 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" containerID="cri-o://aef199335db493a11ae4c78f9987c13086b9bab1aa32bd3c6b4eaf6d7b6c66fd" gracePeriod=25 Oct 30 22:32:30.538134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:30.538074 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" containerID="cri-o://c877e9af1086d892790d8415382a7eae963d937798891bdd428eca55fde4f52d" gracePeriod=25 Oct 30 22:32:31.026644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:31.026600 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg"] Oct 30 22:32:31.031256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:31.031229 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7cd933-trrhg"] Oct 30 22:32:31.035952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:31.035925 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4"] Oct 30 22:32:31.039747 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:31.039724 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f7c4e5-prrs4"] Oct 30 22:32:31.369001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:31.368917 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:32:32.285848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:32.285811 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c40c3f-32db-41a8-bff0-01000e6f237a" path="/var/lib/kubelet/pods/35c40c3f-32db-41a8-bff0-01000e6f237a/volumes" Oct 30 22:32:32.286678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:32.286657 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6830ec2a-d76d-4490-ada9-b9633d7b1ed8" path="/var/lib/kubelet/pods/6830ec2a-d76d-4490-ada9-b9633d7b1ed8/volumes" Oct 30 22:32:32.287162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:32.287143 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c"] Oct 30 22:32:32.290576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:32.290555 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9d868-jvl8c"] Oct 30 22:32:34.189507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:34.189474 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a86b155-c470-46f3-8991-dee0b613ba89" path="/var/lib/kubelet/pods/7a86b155-c470-46f3-8991-dee0b613ba89/volumes" Oct 30 22:32:35.387291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:35.377447 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4"] Oct 30 22:32:35.387291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:35.382643 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-28b4bb-jnpp4"] Oct 30 22:32:36.190803 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:36.190772 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff0f379-1563-4ad5-be5c-a6d3db702726" path="/var/lib/kubelet/pods/bff0f379-1563-4ad5-be5c-a6d3db702726/volumes" Oct 30 22:32:36.369698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:36.369654 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" containerID="cri-o://7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78" gracePeriod=25 Oct 30 22:32:38.054894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:38.054855 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c"] Oct 30 22:32:38.059089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:38.059062 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec933107-92c6c"] Oct 30 22:32:38.190845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:38.190811 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a843eae-17dc-41fa-83ed-f9172d4a1e0e" path="/var/lib/kubelet/pods/4a843eae-17dc-41fa-83ed-f9172d4a1e0e/volumes" Oct 30 22:32:39.658675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:39.658627 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:39.659199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:39.658699 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:40.090037 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.089957 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9"] Oct 30 22:32:40.094241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.094219 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a1ea1e-q2gx9"] Oct 30 22:32:40.203800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.203763 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f78f34-0320-4913-8e1f-81c48e74e9c4" path="/var/lib/kubelet/pods/03f78f34-0320-4913-8e1f-81c48e74e9c4/volumes" Oct 30 22:32:40.809608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.809517 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd"] Oct 30 22:32:40.810638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810611 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03f78f34-0320-4913-8e1f-81c48e74e9c4" containerName="revoke-keys" Oct 30 22:32:40.810638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810632 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f78f34-0320-4913-8e1f-81c48e74e9c4" containerName="revoke-keys" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810657 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810663 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810683 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810688 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810829 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:40.810874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.810847 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f78f34-0320-4913-8e1f-81c48e74e9c4" containerName="revoke-keys" Oct 30 22:32:40.815664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.815645 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.822584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.822565 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:40.823242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.822815 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:40.823242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.822895 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:32:40.823242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.822954 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:40.825503 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.825479 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd"] Oct 30 22:32:40.866418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.866388 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps24b\" (UniqueName: \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.866607 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.866586 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.866656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.866644 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.967906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.967872 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps24b\" (UniqueName: \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.968093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.968047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.968165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.968106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.968767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.968744 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.970786 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.970760 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:40.984811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:40.984791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps24b\" (UniqueName: \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:41.134153 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:41.134120 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:41.461928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:41.461892 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd"] Oct 30 22:32:41.462850 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:41.462822 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e444ad_8427_404c_929b_83c5ad16157e.slice/crio-30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153 WatchSource:0}: Error finding container 30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153: Status 404 returned error can't find the container with id 30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153 Oct 30 22:32:41.550128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:41.550084 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" event={"ID":"a3e444ad-8427-404c-929b-83c5ad16157e","Type":"ContainerStarted","Data":"30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153"} Oct 30 22:32:42.562171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:42.562139 5036 generic.go:356] "Generic (PLEG): container finished" podID="a3e444ad-8427-404c-929b-83c5ad16157e" containerID="5dc491351c81ab010bb5d538744851b7e8b9fc2649c03e0e458e8ae12ff3b1c8" exitCode=0 Oct 30 22:32:42.562629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:42.562186 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" event={"ID":"a3e444ad-8427-404c-929b-83c5ad16157e","Type":"ContainerDied","Data":"5dc491351c81ab010bb5d538744851b7e8b9fc2649c03e0e458e8ae12ff3b1c8"} Oct 30 22:32:43.407122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.406826 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:32:43.407122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.407069 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" containerID="cri-o://baeac5fba68dc44b403da9beec9dc59042f194791237fcfaf0e2f0c25856b7bc" gracePeriod=30 Oct 30 22:32:43.809930 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.809740 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:43.845280 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.845238 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:43.927100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.927066 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\") pod \"a3e444ad-8427-404c-929b-83c5ad16157e\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " Oct 30 22:32:43.927320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.927214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\") pod \"a3e444ad-8427-404c-929b-83c5ad16157e\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " Oct 30 22:32:43.927320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.927252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\" (UniqueName: \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\") pod \"a3e444ad-8427-404c-929b-83c5ad16157e\" (UID: \"a3e444ad-8427-404c-929b-83c5ad16157e\") " Oct 30 22:32:43.927493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.927462 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca" pod "a3e444ad-8427-404c-929b-83c5ad16157e" (UID: "a3e444ad-8427-404c-929b-83c5ad16157e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:43.927991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.927900 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:43.929570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.929542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca" pod "a3e444ad-8427-404c-929b-83c5ad16157e" (UID: "a3e444ad-8427-404c-929b-83c5ad16157e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:43.929756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:43.929737 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b" pod "a3e444ad-8427-404c-929b-83c5ad16157e" (UID: "a3e444ad-8427-404c-929b-83c5ad16157e"). InnerVolumeSpecName "kube-api-access-ps24b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:44.029568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.029495 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e444ad-8427-404c-929b-83c5ad16157e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:44.029568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.029530 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps24b\" (UniqueName: \"kubernetes.io/projected/a3e444ad-8427-404c-929b-83c5ad16157e-kube-api-access-ps24b\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:44.110544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.110515 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr"] Oct 30 22:32:44.111288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.111256 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a3e444ad-8427-404c-929b-83c5ad16157e" containerName="revoke-keys" Oct 30 22:32:44.111360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.111289 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e444ad-8427-404c-929b-83c5ad16157e" containerName="revoke-keys" Oct 30 22:32:44.111408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.111399 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e444ad-8427-404c-929b-83c5ad16157e" containerName="revoke-keys" Oct 30 22:32:44.111445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.111416 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbdf196-4f1a-4542-80d9-b83dafc230b1" containerName="istio-proxy" Oct 30 22:32:44.115389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.115371 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.130937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.130909 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.131081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.130962 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.131180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.131157 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7g7\" (UniqueName: \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.198538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.198503 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr"] Oct 30 22:32:44.232059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.232030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.232248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.232075 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.232248 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.232167 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7g7\" (UniqueName: \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.232865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.232833 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.234657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.234631 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.255277 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.255249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7g7\" (UniqueName: \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\") pod \"maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.443799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.443760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:44.650642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.650608 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" event={"ID":"a3e444ad-8427-404c-929b-83c5ad16157e","Type":"ContainerDied","Data":"30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153"} Oct 30 22:32:44.650839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.650665 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30500be636366f948f9aabec60c895201bb290d071f3c9aea4ebb5503b811153" Oct 30 22:32:44.650839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.650627 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd" Oct 30 22:32:44.694163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.694083 5036 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:44.694163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.694137 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:44.733696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:44.733672 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr"] Oct 30 22:32:44.734309 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:44.734258 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858ec50c_3a05_49b3_ac34_0f74ddc80ab6.slice/crio-6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e WatchSource:0}: Error finding container 6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e: Status 404 returned error can't find the container with id 6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e Oct 30 22:32:45.663168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:45.663131 5036 generic.go:356] "Generic (PLEG): container finished" podID="858ec50c-3a05-49b3-ac34-0f74ddc80ab6" containerID="7a5ae6e697ef3807bec2472baa8f8f5cedfc8f22d531dfd94f50020492e01db4" exitCode=0 Oct 30 22:32:45.663639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:45.663232 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" event={"ID":"858ec50c-3a05-49b3-ac34-0f74ddc80ab6","Type":"ContainerDied","Data":"7a5ae6e697ef3807bec2472baa8f8f5cedfc8f22d531dfd94f50020492e01db4"} Oct 30 22:32:45.663639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:45.663303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" event={"ID":"858ec50c-3a05-49b3-ac34-0f74ddc80ab6","Type":"ContainerStarted","Data":"6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e"} Oct 30 22:32:46.852331 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.852301 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:46.877467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.877440 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\") pod \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " Oct 30 22:32:46.877629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.877478 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\") pod \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " Oct 30 22:32:46.877629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.877596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\" (UniqueName: \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\") pod \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\" (UID: \"858ec50c-3a05-49b3-ac34-0f74ddc80ab6\") " Oct 30 22:32:46.877914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.877887 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca" pod "858ec50c-3a05-49b3-ac34-0f74ddc80ab6" (UID: "858ec50c-3a05-49b3-ac34-0f74ddc80ab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:46.878365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.878316 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:46.880407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.880378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7" pod "858ec50c-3a05-49b3-ac34-0f74ddc80ab6" (UID: "858ec50c-3a05-49b3-ac34-0f74ddc80ab6"). InnerVolumeSpecName "kube-api-access-7x7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:46.880407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.880393 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca" pod "858ec50c-3a05-49b3-ac34-0f74ddc80ab6" (UID: "858ec50c-3a05-49b3-ac34-0f74ddc80ab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:46.979665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.979587 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:46.979665 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:46.979614 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7g7\" (UniqueName: \"kubernetes.io/projected/858ec50c-3a05-49b3-ac34-0f74ddc80ab6-kube-api-access-7x7g7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:47.394827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:47.394796 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:32:47.397351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:47.397317 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" containerID="cri-o://fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be" gracePeriod=30 Oct 30 22:32:47.687355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:47.687243 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" Oct 30 22:32:47.687355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:47.687303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr" event={"ID":"858ec50c-3a05-49b3-ac34-0f74ddc80ab6","Type":"ContainerDied","Data":"6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e"} Oct 30 22:32:47.687355 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:47.687353 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a88fe97cb0aaa9e3909c832e9646d12cb9ed686bfdca237eb150c2881c74b6e" Oct 30 22:32:48.006593 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.006519 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b"] Oct 30 22:32:48.007245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.007231 5036 cpu_manager.go:414] "containerMap: removing container" podUID="858ec50c-3a05-49b3-ac34-0f74ddc80ab6" containerName="revoke-keys" Oct 30 22:32:48.007315 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.007246 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858ec50c-3a05-49b3-ac34-0f74ddc80ab6" containerName="revoke-keys" Oct 30 22:32:48.007394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.007384 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="858ec50c-3a05-49b3-ac34-0f74ddc80ab6" containerName="revoke-keys" Oct 30 22:32:48.011080 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.011065 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.016180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.016142 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:48.016588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.016568 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:48.016976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.016953 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:48.017094 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.017035 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:32:48.022633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.022586 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b"] Oct 30 22:32:48.100323 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:48.100287 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:32:48.122161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.120350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.122161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.120501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.122161 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.120605 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvgj\" (UniqueName: \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.221872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.221841 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvgj\" (UniqueName: \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.222066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.222025 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.222131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.222095 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.222731 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.222708 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.224838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.224816 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.235605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.235577 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvgj\" (UniqueName: \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.331516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.331433 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:48.632642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.632608 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b"] Oct 30 22:32:48.634024 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:48.633979 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4722f7_0ab7_48a5_ae00_a37822d098db.slice/crio-f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9 WatchSource:0}: Error finding container f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9: Status 404 returned error can't find the container with id f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9 Oct 30 22:32:48.725724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.725678 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" event={"ID":"9b4722f7-0ab7-48a5-ae00-a37822d098db","Type":"ContainerStarted","Data":"f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9"} Oct 30 22:32:48.733126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.733066 5036 generic.go:356] "Generic (PLEG): container finished" podID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerID="baeac5fba68dc44b403da9beec9dc59042f194791237fcfaf0e2f0c25856b7bc" exitCode=0 Oct 30 22:32:48.733319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.733133 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerDied","Data":"baeac5fba68dc44b403da9beec9dc59042f194791237fcfaf0e2f0c25856b7bc"} Oct 30 22:32:48.733319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.733207 5036 scope.go:122] "RemoveContainer" containerID="4d14ca2a54c8528d009e7cd8975093a70a298de7052ffbc858b8a70a9a63c368" Oct 30 22:32:48.742207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.742180 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.860788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.860845 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.860972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861105 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861139 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.861238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861167 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861511 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861590 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.861685 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\") pod \"fca4cc46-0c3b-4ac7-854c-6563870206b0\" (UID: \"fca4cc46-0c3b-4ac7-854c-6563870206b0\") " Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.865904 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.866599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.867557 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:48.862003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.869222 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:48.870971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.870943 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:48.880400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880286 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.880400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880327 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.880400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880344 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca4cc46-0c3b-4ac7-854c-6563870206b0-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.880400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880360 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.880400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880374 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.880809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.880655 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 22:32:48.885313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.885278 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:48.885479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.885457 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:48.886011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.885985 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:48.887474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.887447 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986" pod "fca4cc46-0c3b-4ac7-854c-6563870206b0" (UID: "fca4cc46-0c3b-4ac7-854c-6563870206b0"). InnerVolumeSpecName "kube-api-access-s5986". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:48.982247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.982215 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.982247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.982243 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.982504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.982256 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5986\" (UniqueName: \"kubernetes.io/projected/fca4cc46-0c3b-4ac7-854c-6563870206b0-kube-api-access-s5986\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:48.982504 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:48.982280 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca4cc46-0c3b-4ac7-854c-6563870206b0-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:49.657840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.657805 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:49.658288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.658160 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:49.744587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.744552 5036 generic.go:356] "Generic (PLEG): container finished" podID="9b4722f7-0ab7-48a5-ae00-a37822d098db" containerID="80483077b0b310ca2a3c0d1cc7df32be155ff44ace4587916840ac37fa7563bd" exitCode=0 Oct 30 22:32:49.744796 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.744646 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" event={"ID":"9b4722f7-0ab7-48a5-ae00-a37822d098db","Type":"ContainerDied","Data":"80483077b0b310ca2a3c0d1cc7df32be155ff44ace4587916840ac37fa7563bd"} Oct 30 22:32:49.746523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.746465 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" Oct 30 22:32:49.746523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.746493 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd" event={"ID":"fca4cc46-0c3b-4ac7-854c-6563870206b0","Type":"ContainerDied","Data":"2507a1d0f92e5a485bf46f66d20c28621f1ab88811b9934d5274df090130fc8e"} Oct 30 22:32:49.746708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.746554 5036 scope.go:122] "RemoveContainer" containerID="baeac5fba68dc44b403da9beec9dc59042f194791237fcfaf0e2f0c25856b7bc" Oct 30 22:32:49.796871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.796832 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:32:49.805440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:49.805411 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-58pnd"] Oct 30 22:32:50.031634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.031598 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w"] Oct 30 22:32:50.035876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.035850 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be5ba7-l9d6w"] Oct 30 22:32:50.190280 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.190230 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d26225-aa3f-4a5f-a4f2-3c116cafe885" path="/var/lib/kubelet/pods/54d26225-aa3f-4a5f-a4f2-3c116cafe885/volumes" Oct 30 22:32:50.190848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.190832 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" path="/var/lib/kubelet/pods/fca4cc46-0c3b-4ac7-854c-6563870206b0/volumes" Oct 30 22:32:50.750888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.750851 5036 scope.go:122] "RemoveContainer" containerID="fb1541c32dfa1b42c4d254f936b66e9a31634d643fb67c43aeda4f687ebcecba" Oct 30 22:32:50.764028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.763932 5036 generic.go:356] "Generic (PLEG): container finished" podID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerID="aef199335db493a11ae4c78f9987c13086b9bab1aa32bd3c6b4eaf6d7b6c66fd" exitCode=0 Oct 30 22:32:50.764028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.764008 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" event={"ID":"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21","Type":"ContainerDied","Data":"aef199335db493a11ae4c78f9987c13086b9bab1aa32bd3c6b4eaf6d7b6c66fd"} Oct 30 22:32:50.766390 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.766006 5036 generic.go:356] "Generic (PLEG): container finished" podID="8f376211-2a5d-475b-adfa-394b499e2f52" containerID="c877e9af1086d892790d8415382a7eae963d937798891bdd428eca55fde4f52d" exitCode=0 Oct 30 22:32:50.766533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.766510 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" event={"ID":"8f376211-2a5d-475b-adfa-394b499e2f52","Type":"ContainerDied","Data":"c877e9af1086d892790d8415382a7eae963d937798891bdd428eca55fde4f52d"} Oct 30 22:32:50.771136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.770891 5036 scope.go:122] "RemoveContainer" containerID="19efd60663b57bb22a63d39b9d2f7efe9bba6f7a4ad439a6cb88348e311be77a" Oct 30 22:32:50.877477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.877422 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:32:50.955641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.940358 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\" (UniqueName: \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\") pod \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " Oct 30 22:32:50.955641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.940450 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\") pod \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\" (UID: \"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21\") " Oct 30 22:32:50.955641 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.942072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume" pod "e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" (UID: "e52e5962-6fe0-4f8b-9a72-db5dc01c6a21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:50.963827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.963794 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:32:50.969571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:50.969517 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf" pod "e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" (UID: "e52e5962-6fe0-4f8b-9a72-db5dc01c6a21"). InnerVolumeSpecName "kube-api-access-jvrxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:51.065221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.065137 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\" (UniqueName: \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\") pod \"8f376211-2a5d-475b-adfa-394b499e2f52\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " Oct 30 22:32:51.065221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.065217 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\") pod \"8f376211-2a5d-475b-adfa-394b499e2f52\" (UID: \"8f376211-2a5d-475b-adfa-394b499e2f52\") " Oct 30 22:32:51.065952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.065917 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume" pod "8f376211-2a5d-475b-adfa-394b499e2f52" (UID: "8f376211-2a5d-475b-adfa-394b499e2f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:51.066882 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.066858 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrxf\" (UniqueName: \"kubernetes.io/projected/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-kube-api-access-jvrxf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.067063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.067045 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.067186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.067173 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f376211-2a5d-475b-adfa-394b499e2f52-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.069604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.069557 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56" pod "8f376211-2a5d-475b-adfa-394b499e2f52" (UID: "8f376211-2a5d-475b-adfa-394b499e2f52"). InnerVolumeSpecName "kube-api-access-m5f56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:51.086176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.086145 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:51.168819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.168780 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\" (UniqueName: \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\") pod \"9b4722f7-0ab7-48a5-ae00-a37822d098db\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " Oct 30 22:32:51.169035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.168904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\") pod \"9b4722f7-0ab7-48a5-ae00-a37822d098db\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " Oct 30 22:32:51.169035 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.168971 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\") pod \"9b4722f7-0ab7-48a5-ae00-a37822d098db\" (UID: \"9b4722f7-0ab7-48a5-ae00-a37822d098db\") " Oct 30 22:32:51.169656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.169620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca" pod "9b4722f7-0ab7-48a5-ae00-a37822d098db" (UID: "9b4722f7-0ab7-48a5-ae00-a37822d098db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:51.169903 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.169880 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.169986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.169911 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5f56\" (UniqueName: \"kubernetes.io/projected/8f376211-2a5d-475b-adfa-394b499e2f52-kube-api-access-m5f56\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.171443 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.171408 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj" pod "9b4722f7-0ab7-48a5-ae00-a37822d098db" (UID: "9b4722f7-0ab7-48a5-ae00-a37822d098db"). InnerVolumeSpecName "kube-api-access-5dvgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:51.171743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.171723 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca" pod "9b4722f7-0ab7-48a5-ae00-a37822d098db" (UID: "9b4722f7-0ab7-48a5-ae00-a37822d098db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:51.293293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.280561 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvgj\" (UniqueName: \"kubernetes.io/projected/9b4722f7-0ab7-48a5-ae00-a37822d098db-kube-api-access-5dvgj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.293293 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.280604 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4722f7-0ab7-48a5-ae00-a37822d098db-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:51.775708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.775678 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" Oct 30 22:32:51.776202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.775702 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8" event={"ID":"e52e5962-6fe0-4f8b-9a72-db5dc01c6a21","Type":"ContainerDied","Data":"9a358b9f6b4932fc56f0a830489bba5e7f3cf935010c2c3e049a496f7be07403"} Oct 30 22:32:51.776202 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.775766 5036 scope.go:122] "RemoveContainer" containerID="aef199335db493a11ae4c78f9987c13086b9bab1aa32bd3c6b4eaf6d7b6c66fd" Oct 30 22:32:51.777543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.777515 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" event={"ID":"9b4722f7-0ab7-48a5-ae00-a37822d098db","Type":"ContainerDied","Data":"f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9"} Oct 30 22:32:51.777637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.777569 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91d42a4bb2807c29a2edc83afab1b067097362e94230f9008ab10d9709618e9" Oct 30 22:32:51.777637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.777527 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b" Oct 30 22:32:51.779626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.779600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" event={"ID":"8f376211-2a5d-475b-adfa-394b499e2f52","Type":"ContainerDied","Data":"1e3c7d191ffb9574b9b0b68c6d0b6aab92bbff74ce1a0fb97401fa25076ba86e"} Oct 30 22:32:51.779743 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.779614 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz" Oct 30 22:32:51.788927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.788908 5036 scope.go:122] "RemoveContainer" containerID="c877e9af1086d892790d8415382a7eae963d937798891bdd428eca55fde4f52d" Oct 30 22:32:51.824246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.824217 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:32:51.828294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.828254 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9d868-695688b885-z6pw8"] Oct 30 22:32:51.836043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.836021 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:32:51.842038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:51.842016 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9d868-7b9f54b77f-5r6kz"] Oct 30 22:32:52.189008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.188975 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" path="/var/lib/kubelet/pods/8f376211-2a5d-475b-adfa-394b499e2f52/volumes" Oct 30 22:32:52.189436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.189422 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" path="/var/lib/kubelet/pods/e52e5962-6fe0-4f8b-9a72-db5dc01c6a21/volumes" Oct 30 22:32:52.735180 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.735149 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:32:52.792104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.792010 5036 generic.go:356] "Generic (PLEG): container finished" podID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerID="fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be" exitCode=0 Oct 30 22:32:52.792104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.792073 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" Oct 30 22:32:52.792104 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.792079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerDied","Data":"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be"} Oct 30 22:32:52.792697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.792173 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj" event={"ID":"cec71c40-29db-4de4-b757-3c5d5a6573e4","Type":"ContainerDied","Data":"b74187c614a800d5e8942f1268ed60e47989c1c16b303b675455325128a945a4"} Oct 30 22:32:52.792697 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.792204 5036 scope.go:122] "RemoveContainer" containerID="fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be" Oct 30 22:32:52.804779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.804760 5036 scope.go:122] "RemoveContainer" containerID="cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74" Oct 30 22:32:52.816745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.816718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.816958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.816939 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817036 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.816987 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817036 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.817127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817122 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817255 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.817368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817339 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817402 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\") pod \"cec71c40-29db-4de4-b757-3c5d5a6573e4\" (UID: \"cec71c40-29db-4de4-b757-3c5d5a6573e4\") " Oct 30 22:32:52.817632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.817604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:52.821347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.818020 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.821347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.818176 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec71c40-29db-4de4-b757-3c5d5a6573e4-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.821347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.818198 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.821347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.818212 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.821347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.818228 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.822834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.822552 5036 scope.go:122] "RemoveContainer" containerID="fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be" Oct 30 22:32:52.823354 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:52.823028 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be\": container with ID starting with fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be not found: ID does not exist" containerID="fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be" Oct 30 22:32:52.823354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.823066 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be"} err="failed to get container status \"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be\": rpc error: code = NotFound desc = could not find container \"fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be\": container with ID starting with fb21f3ce5b9d78b2969325c2c3d0f9acfcfe9547d1bcb5d10bd9dee55040f2be not found: ID does not exist" Oct 30 22:32:52.823354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.823092 5036 scope.go:122] "RemoveContainer" containerID="cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74" Oct 30 22:32:52.823582 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:52.823442 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74\": container with ID starting with cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74 not found: ID does not exist" containerID="cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74" Oct 30 22:32:52.823582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.823477 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74"} err="failed to get container status \"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74\": rpc error: code = NotFound desc = could not find container \"cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74\": container with ID starting with cf11b452ed11ef43f2de1cd2d37969ca2a3f93c99176257741c39958605f1f74 not found: ID does not exist" Oct 30 22:32:52.823791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.823763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.824694 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.824672 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:52.824826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.824801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:52.824926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.824908 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.826484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.826458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz" pod "cec71c40-29db-4de4-b757-3c5d5a6573e4" (UID: "cec71c40-29db-4de4-b757-3c5d5a6573e4"). InnerVolumeSpecName "kube-api-access-fqjnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:52.920460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.920417 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.920460 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.920460 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.920655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.920478 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.920655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.920491 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.920655 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:52.920505 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjnz\" (UniqueName: \"kubernetes.io/projected/cec71c40-29db-4de4-b757-3c5d5a6573e4-kube-api-access-fqjnz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:53.144420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:53.144374 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:32:53.156019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:53.155971 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a1ea1e-openshift-default-64dbd9f697-96zxj"] Oct 30 22:32:53.809519 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:53.809483 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:32:54.188944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:54.188913 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" path="/var/lib/kubelet/pods/cec71c40-29db-4de4-b757-3c5d5a6573e4/volumes" Oct 30 22:32:54.243680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:54.243645 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 22:32:54.263922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:54.263863 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 22:32:55.694005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.693967 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5"] Oct 30 22:32:55.695040 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695017 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.695093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695043 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.695093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695075 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.695093 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695083 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695119 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695128 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695148 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695156 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695183 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9b4722f7-0ab7-48a5-ae00-a37822d098db" containerName="revoke-keys" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695190 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4722f7-0ab7-48a5-ae00-a37822d098db" containerName="revoke-keys" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695204 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695212 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695221 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" Oct 30 22:32:55.695238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695230 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695419 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695437 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695451 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4722f7-0ab7-48a5-ae00-a37822d098db" containerName="revoke-keys" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695462 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52e5962-6fe0-4f8b-9a72-db5dc01c6a21" containerName="payload-processing" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695472 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f376211-2a5d-475b-adfa-394b499e2f52" containerName="payload-pre-processing" Oct 30 22:32:55.695736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.695487 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca4cc46-0c3b-4ac7-854c-6563870206b0" containerName="istio-proxy" Oct 30 22:32:55.701661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.701639 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc"] Oct 30 22:32:55.701804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.701772 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.703388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.703362 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec71c40-29db-4de4-b757-3c5d5a6573e4" containerName="istio-proxy" Oct 30 22:32:55.708092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5"] Oct 30 22:32:55.708220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708174 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.710101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708630 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:55.710101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708735 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:55.710101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708823 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:32:55.710101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.708868 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:55.756560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.722548 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc"] Oct 30 22:32:55.797007 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.796967 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.797200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.797046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.797200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.797169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695bs\" (UniqueName: \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.797200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.797196 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.797414 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.797368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnnk\" (UniqueName: \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.797454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.797433 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.898812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.898767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.899008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.898879 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695bs\" (UniqueName: \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.899008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.898908 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.899162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.899071 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnnk\" (UniqueName: \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.899162 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.899149 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.899295 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.899258 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.899626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.899600 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.899683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.899664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.901756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.901725 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.901927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.901908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:55.911998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.911964 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-695bs\" (UniqueName: \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\") pod \"maas-api-revoke-sub-52d023c1-j9wwc\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:55.912108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:55.912054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnnk\" (UniqueName: \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\") pod \"maas-api-revoke-sub-3a16d9ed-t45n5\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:56.056028 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.049740 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:56.064027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.063724 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:56.376452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.376420 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc"] Oct 30 22:32:56.376702 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:56.376667 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd974d719_c312_426b_b99b_c6082de3a3d8.slice/crio-d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef WatchSource:0}: Error finding container d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef: Status 404 returned error can't find the container with id d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef Oct 30 22:32:56.382215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.382189 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5"] Oct 30 22:32:56.383324 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:32:56.383298 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff19f61_6c90_45c6_aecc_d07b3fd3adb3.slice/crio-8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332 WatchSource:0}: Error finding container 8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332: Status 404 returned error can't find the container with id 8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332 Oct 30 22:32:56.642891 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.642868 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:32:56.814462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.814419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\") pod \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " Oct 30 22:32:56.814952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.814560 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\" (UniqueName: \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\") pod \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\" (UID: \"a80a64c9-ef5d-4057-9b42-0d5050fb7419\") " Oct 30 22:32:56.814952 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.814880 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume" pod "a80a64c9-ef5d-4057-9b42-0d5050fb7419" (UID: "a80a64c9-ef5d-4057-9b42-0d5050fb7419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:56.815419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.815397 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a80a64c9-ef5d-4057-9b42-0d5050fb7419-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:56.817918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.817886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp" pod "a80a64c9-ef5d-4057-9b42-0d5050fb7419" (UID: "a80a64c9-ef5d-4057-9b42-0d5050fb7419"). InnerVolumeSpecName "kube-api-access-z29gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:56.841252 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.841216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" event={"ID":"fff19f61-6c90-45c6-aecc-d07b3fd3adb3","Type":"ContainerStarted","Data":"9e459752a65293f1ee8980135a438c36e3292931a7cb74d7a893e519f131ddc9"} Oct 30 22:32:56.841425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.841321 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" event={"ID":"fff19f61-6c90-45c6-aecc-d07b3fd3adb3","Type":"ContainerStarted","Data":"8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332"} Oct 30 22:32:56.843374 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.843343 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" event={"ID":"d974d719-c312-426b-b99b-c6082de3a3d8","Type":"ContainerStarted","Data":"8146755d29b01af5b6bfb7988752107efb4af3da77c6cd5ebe25584ff9780c84"} Oct 30 22:32:56.843498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.843403 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" event={"ID":"d974d719-c312-426b-b99b-c6082de3a3d8","Type":"ContainerStarted","Data":"d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef"} Oct 30 22:32:56.845066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.845041 5036 generic.go:356] "Generic (PLEG): container finished" podID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerID="7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78" exitCode=0 Oct 30 22:32:56.845193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.845092 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" Oct 30 22:32:56.845193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.845119 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" event={"ID":"a80a64c9-ef5d-4057-9b42-0d5050fb7419","Type":"ContainerDied","Data":"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78"} Oct 30 22:32:56.845193 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.845180 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh" event={"ID":"a80a64c9-ef5d-4057-9b42-0d5050fb7419","Type":"ContainerDied","Data":"74df7fe85640f19ff9039710be0dde709ab3444def871c49ea6550040212e4e5"} Oct 30 22:32:56.845412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.845213 5036 scope.go:122] "RemoveContainer" containerID="7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78" Oct 30 22:32:56.856992 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.856974 5036 scope.go:122] "RemoveContainer" containerID="7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78" Oct 30 22:32:56.857356 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:32:56.857332 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78\": container with ID starting with 7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78 not found: ID does not exist" containerID="7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78" Oct 30 22:32:56.857436 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.857360 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78"} err="failed to get container status \"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78\": rpc error: code = NotFound desc = could not find container \"7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78\": container with ID starting with 7b7c6ac46f38917a6091c23b7c2a98268f1bf5f18a7bf94c82fee01bcf750b78 not found: ID does not exist" Oct 30 22:32:56.886066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.886016 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" podStartSLOduration=1.8860023529999999 podStartE2EDuration="1.886002353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:56.882509934 +0000 UTC m=+3200.832464891" watchObservedRunningTime="2026-10-30 22:32:56.886002353 +0000 UTC m=+3200.835957299" Oct 30 22:32:56.917156 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.917013 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z29gp\" (UniqueName: \"kubernetes.io/projected/a80a64c9-ef5d-4057-9b42-0d5050fb7419-kube-api-access-z29gp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:56.917505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.917467 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" podStartSLOduration=1.917454345 podStartE2EDuration="1.917454345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:56.91569873 +0000 UTC m=+3200.865653680" watchObservedRunningTime="2026-10-30 22:32:56.917454345 +0000 UTC m=+3200.867409303" Oct 30 22:32:56.931586 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.931551 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:32:56.938430 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:56.938405 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-28b4bb-9c8644f85-2kwhh"] Oct 30 22:32:57.859774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:57.859688 5036 generic.go:356] "Generic (PLEG): container finished" podID="fff19f61-6c90-45c6-aecc-d07b3fd3adb3" containerID="9e459752a65293f1ee8980135a438c36e3292931a7cb74d7a893e519f131ddc9" exitCode=0 Oct 30 22:32:57.860230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:57.859784 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" event={"ID":"fff19f61-6c90-45c6-aecc-d07b3fd3adb3","Type":"ContainerDied","Data":"9e459752a65293f1ee8980135a438c36e3292931a7cb74d7a893e519f131ddc9"} Oct 30 22:32:57.861546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:57.861523 5036 generic.go:356] "Generic (PLEG): container finished" podID="d974d719-c312-426b-b99b-c6082de3a3d8" containerID="8146755d29b01af5b6bfb7988752107efb4af3da77c6cd5ebe25584ff9780c84" exitCode=0 Oct 30 22:32:57.861661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:57.861560 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" event={"ID":"d974d719-c312-426b-b99b-c6082de3a3d8","Type":"ContainerDied","Data":"8146755d29b01af5b6bfb7988752107efb4af3da77c6cd5ebe25584ff9780c84"} Oct 30 22:32:58.140330 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:58.140293 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:32:58.159029 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:58.158990 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:32:58.198971 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:58.197387 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" path="/var/lib/kubelet/pods/a80a64c9-ef5d-4057-9b42-0d5050fb7419/volumes" Oct 30 22:32:59.113538 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.113505 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:59.173658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.173629 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\" (UniqueName: \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\") pod \"d974d719-c312-426b-b99b-c6082de3a3d8\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " Oct 30 22:32:59.173883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.173866 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\") pod \"d974d719-c312-426b-b99b-c6082de3a3d8\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " Oct 30 22:32:59.174054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.173953 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\") pod \"d974d719-c312-426b-b99b-c6082de3a3d8\" (UID: \"d974d719-c312-426b-b99b-c6082de3a3d8\") " Oct 30 22:32:59.174558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.174500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca" pod "d974d719-c312-426b-b99b-c6082de3a3d8" (UID: "d974d719-c312-426b-b99b-c6082de3a3d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:59.174744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.174710 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.176751 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.176720 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs" pod "d974d719-c312-426b-b99b-c6082de3a3d8" (UID: "d974d719-c312-426b-b99b-c6082de3a3d8"). InnerVolumeSpecName "kube-api-access-695bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:59.176894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.176866 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca" pod "d974d719-c312-426b-b99b-c6082de3a3d8" (UID: "d974d719-c312-426b-b99b-c6082de3a3d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:59.204989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.204966 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:59.244896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.244852 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" containerID="cri-o://62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534" gracePeriod=25 Oct 30 22:32:59.264611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.264544 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" containerID="cri-o://b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052" gracePeriod=25 Oct 30 22:32:59.275615 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.275588 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\" (UniqueName: \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\") pod \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " Oct 30 22:32:59.275792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.275774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\") pod \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " Oct 30 22:32:59.275907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.275890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\") pod \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\" (UID: \"fff19f61-6c90-45c6-aecc-d07b3fd3adb3\") " Oct 30 22:32:59.276250 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.276223 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca" pod "fff19f61-6c90-45c6-aecc-d07b3fd3adb3" (UID: "fff19f61-6c90-45c6-aecc-d07b3fd3adb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:59.276570 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.276548 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-695bs\" (UniqueName: \"kubernetes.io/projected/d974d719-c312-426b-b99b-c6082de3a3d8-kube-api-access-695bs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.276670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.276577 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.276670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.276593 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d974d719-c312-426b-b99b-c6082de3a3d8-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.277855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.277829 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk" pod "fff19f61-6c90-45c6-aecc-d07b3fd3adb3" (UID: "fff19f61-6c90-45c6-aecc-d07b3fd3adb3"). InnerVolumeSpecName "kube-api-access-2rnnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:59.278057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.278026 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca" pod "fff19f61-6c90-45c6-aecc-d07b3fd3adb3" (UID: "fff19f61-6c90-45c6-aecc-d07b3fd3adb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:59.377771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.377738 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnnk\" (UniqueName: \"kubernetes.io/projected/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-kube-api-access-2rnnk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.377771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.377770 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff19f61-6c90-45c6-aecc-d07b3fd3adb3-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:59.881206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.880524 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" Oct 30 22:32:59.881206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.880543 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a16d9ed-t45n5" event={"ID":"fff19f61-6c90-45c6-aecc-d07b3fd3adb3","Type":"ContainerDied","Data":"8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332"} Oct 30 22:32:59.881206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.880597 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8771671ede14a215f63fe381cf6f4736f4abe7efa7c6db0abe1b3b2fefa0e332" Oct 30 22:32:59.882294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.882256 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" Oct 30 22:32:59.882475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.882315 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d023c1-j9wwc" event={"ID":"d974d719-c312-426b-b99b-c6082de3a3d8","Type":"ContainerDied","Data":"d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef"} Oct 30 22:32:59.882475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:32:59.882360 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d203bf003cf8816e2a6f75cee5b0c1e170102446167fc86c29499f2849043aef" Oct 30 22:33:01.019825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.019794 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx"] Oct 30 22:33:01.020528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020512 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fff19f61-6c90-45c6-aecc-d07b3fd3adb3" containerName="revoke-keys" Oct 30 22:33:01.020528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020529 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff19f61-6c90-45c6-aecc-d07b3fd3adb3" containerName="revoke-keys" Oct 30 22:33:01.020635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020590 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" Oct 30 22:33:01.020635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020595 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" Oct 30 22:33:01.020635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020606 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d974d719-c312-426b-b99b-c6082de3a3d8" containerName="revoke-keys" Oct 30 22:33:01.020635 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020611 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d974d719-c312-426b-b99b-c6082de3a3d8" containerName="revoke-keys" Oct 30 22:33:01.020776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020705 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff19f61-6c90-45c6-aecc-d07b3fd3adb3" containerName="revoke-keys" Oct 30 22:33:01.020776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020721 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80a64c9-ef5d-4057-9b42-0d5050fb7419" containerName="payload-processing" Oct 30 22:33:01.020776 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.020728 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d974d719-c312-426b-b99b-c6082de3a3d8" containerName="revoke-keys" Oct 30 22:33:01.024633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.024618 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.029294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.027820 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:01.029294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.028518 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:01.029294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.028566 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:01.029294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.028622 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:01.039144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.039120 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx"] Oct 30 22:33:01.098558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.098530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.098749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.098588 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85vz7\" (UniqueName: \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.098797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.098747 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.199946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.199915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.200167 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.200140 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.200257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.200199 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85vz7\" (UniqueName: \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.200762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.200735 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.202573 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.202552 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.212520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.212494 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85vz7\" (UniqueName: \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\") pod \"maas-api-revoke-sub-085509f7-zd2gx\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.344648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.344554 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:01.639404 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.639370 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx"] Oct 30 22:33:01.643768 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:01.643731 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548c7370_0a93_4c92_a9d6_859dcef4af0a.slice/crio-5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305 WatchSource:0}: Error finding container 5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305: Status 404 returned error can't find the container with id 5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305 Oct 30 22:33:01.853227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.853194 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd"] Oct 30 22:33:01.857385 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.857360 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5f192-l2ttd"] Oct 30 22:33:01.941824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:01.941735 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" event={"ID":"548c7370-0a93-4c92-a9d6-859dcef4af0a","Type":"ContainerStarted","Data":"5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305"} Oct 30 22:33:02.190090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:02.190056 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e444ad-8427-404c-929b-83c5ad16157e" path="/var/lib/kubelet/pods/a3e444ad-8427-404c-929b-83c5ad16157e/volumes" Oct 30 22:33:02.964590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:02.964553 5036 generic.go:356] "Generic (PLEG): container finished" podID="548c7370-0a93-4c92-a9d6-859dcef4af0a" containerID="f4abb148218d7685596e463dd7b0e19805e9bf3178d181797e23ff30ea48360b" exitCode=0 Oct 30 22:33:02.964778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:02.964637 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" event={"ID":"548c7370-0a93-4c92-a9d6-859dcef4af0a","Type":"ContainerDied","Data":"f4abb148218d7685596e463dd7b0e19805e9bf3178d181797e23ff30ea48360b"} Oct 30 22:33:03.141173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:03.141116 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" containerID="cri-o://83c53512533b378cba3f2d8ead713de4fce173bb436012c82a3dfe7de9e7b934" gracePeriod=25 Oct 30 22:33:03.159874 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:03.159827 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" containerID="cri-o://5a4ebed3ba61c3846effb583552a2088d4ff45bf310f57c86a4a9490bd480018" gracePeriod=25 Oct 30 22:33:03.906117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:03.906069 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" containerID="cri-o://4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3" gracePeriod=45 Oct 30 22:33:03.955639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:03.955606 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 22:33:04.267291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.244758 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:04.412062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.412029 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\") pod \"548c7370-0a93-4c92-a9d6-859dcef4af0a\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " Oct 30 22:33:04.412198 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.412156 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\") pod \"548c7370-0a93-4c92-a9d6-859dcef4af0a\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " Oct 30 22:33:04.412255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.412212 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\" (UniqueName: \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\") pod \"548c7370-0a93-4c92-a9d6-859dcef4af0a\" (UID: \"548c7370-0a93-4c92-a9d6-859dcef4af0a\") " Oct 30 22:33:04.416544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.416488 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca" pod "548c7370-0a93-4c92-a9d6-859dcef4af0a" (UID: "548c7370-0a93-4c92-a9d6-859dcef4af0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:04.418003 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.417971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7" pod "548c7370-0a93-4c92-a9d6-859dcef4af0a" (UID: "548c7370-0a93-4c92-a9d6-859dcef4af0a"). InnerVolumeSpecName "kube-api-access-85vz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:04.418152 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.418111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca" pod "548c7370-0a93-4c92-a9d6-859dcef4af0a" (UID: "548c7370-0a93-4c92-a9d6-859dcef4af0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:04.418516 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.418483 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.456361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.456325 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:33:04.491334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.479978 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b"] Oct 30 22:33:04.491334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.490426 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecfbd6-sf94b"] Oct 30 22:33:04.521294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.520579 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548c7370-0a93-4c92-a9d6-859dcef4af0a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.521294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.520615 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85vz7\" (UniqueName: \"kubernetes.io/projected/548c7370-0a93-4c92-a9d6-859dcef4af0a-kube-api-access-85vz7\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.633287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.629865 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\" (UniqueName: \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.633287 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.629969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.646435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.643035 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.647969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.647923 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "kube-api-access-xc7jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:04.650595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.649863 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.650595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.649999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.650595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650052 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.650595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650077 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\") pod \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\" (UID: \"ad4a8350-47f0-480a-9ac3-6b993bd07c1e\") " Oct 30 22:33:04.650595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.650970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.650970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650910 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7jg\" (UniqueName: \"kubernetes.io/projected/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-kube-api-access-xc7jg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.650970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650933 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.650970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650945 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.650970 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.650957 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.652806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.652785 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:04.653127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.653112 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm" pod "ad4a8350-47f0-480a-9ac3-6b993bd07c1e" (UID: "ad4a8350-47f0-480a-9ac3-6b993bd07c1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.752137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.752063 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.752137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.752092 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a8350-47f0-480a-9ac3-6b993bd07c1e-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.983908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.983877 5036 generic.go:356] "Generic (PLEG): container finished" podID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerID="4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3" exitCode=0 Oct 30 22:33:04.984427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.983941 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" Oct 30 22:33:04.984427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.983965 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" event={"ID":"ad4a8350-47f0-480a-9ac3-6b993bd07c1e","Type":"ContainerDied","Data":"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3"} Oct 30 22:33:04.984427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.984010 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg" event={"ID":"ad4a8350-47f0-480a-9ac3-6b993bd07c1e","Type":"ContainerDied","Data":"c2a8de16a9fa66adf23bd325d715e254e7d90c513a50babcf4b9a8188de19a56"} Oct 30 22:33:04.984427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.984028 5036 scope.go:122] "RemoveContainer" containerID="4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3" Oct 30 22:33:04.985773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.985727 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" event={"ID":"548c7370-0a93-4c92-a9d6-859dcef4af0a","Type":"ContainerDied","Data":"5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305"} Oct 30 22:33:04.985773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.985759 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085509f7-zd2gx" Oct 30 22:33:04.985911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.985765 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c8ea1937dba788f38d5b1698ad3c7625c9033995085efd18f49613f52d90305" Oct 30 22:33:04.998008 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.997986 5036 scope.go:122] "RemoveContainer" containerID="4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3" Oct 30 22:33:04.998389 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:04.998359 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3\": container with ID starting with 4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3 not found: ID does not exist" containerID="4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3" Oct 30 22:33:04.998494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:04.998398 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3"} err="failed to get container status \"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3\": rpc error: code = NotFound desc = could not find container \"4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3\": container with ID starting with 4b40607fca721d9e0127b3c14a639e0245acbd3b3988990aac17a76e49bd25f3 not found: ID does not exist" Oct 30 22:33:05.027112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:05.027038 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 22:33:05.033782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:05.033755 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/premium-w1-0f26c5-kserve-6c898c44b7-jzwvg"] Oct 30 22:33:06.161948 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:06.161907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:06.162370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:06.161907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:06.190736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:06.190704 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4722f7-0ab7-48a5-ae00-a37822d098db" path="/var/lib/kubelet/pods/9b4722f7-0ab7-48a5-ae00-a37822d098db/volumes" Oct 30 22:33:06.191303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:06.191242 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" path="/var/lib/kubelet/pods/ad4a8350-47f0-480a-9ac3-6b993bd07c1e/volumes" Oct 30 22:33:08.541533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:08.541496 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:08.541955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:08.541505 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:09.699518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:09.699322 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:33:09.699953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:09.699576 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" containerID="cri-o://7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5" gracePeriod=30 Oct 30 22:33:10.161947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.161914 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx"] Oct 30 22:33:10.162629 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162614 5036 cpu_manager.go:414] "containerMap: removing container" podUID="548c7370-0a93-4c92-a9d6-859dcef4af0a" containerName="revoke-keys" Oct 30 22:33:10.162683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162631 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548c7370-0a93-4c92-a9d6-859dcef4af0a" containerName="revoke-keys" Oct 30 22:33:10.162683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162650 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" Oct 30 22:33:10.162683 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162655 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" Oct 30 22:33:10.162790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162761 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4a8350-47f0-480a-9ac3-6b993bd07c1e" containerName="main" Oct 30 22:33:10.162790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.162779 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="548c7370-0a93-4c92-a9d6-859dcef4af0a" containerName="revoke-keys" Oct 30 22:33:10.167553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.167524 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.173298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.171177 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:10.173298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.171204 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:10.173298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.171475 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:10.173298 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.171495 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:10.176324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.176294 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx"] Oct 30 22:33:10.278765 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.278733 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.279056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.279023 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pb7s\" (UniqueName: \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.279200 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.279084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.381172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.381128 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pb7s\" (UniqueName: \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.381373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.381194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.381373 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.381352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.382032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.381970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.383857 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.383830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.392790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.392761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pb7s\" (UniqueName: \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.491119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.491039 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:10.892603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:10.892575 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx"] Oct 30 22:33:10.893681 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:10.893648 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828f9550_05b1_486c_b332_fe3d7f4d701b.slice/crio-bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c WatchSource:0}: Error finding container bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c: Status 404 returned error can't find the container with id bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c Oct 30 22:33:11.107380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:11.107331 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" event={"ID":"828f9550-05b1-486c-b332-fe3d7f4d701b","Type":"ContainerStarted","Data":"bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c"} Oct 30 22:33:11.491512 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:11.491477 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr"] Oct 30 22:33:11.495571 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:11.495539 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0f26c5-d6trr"] Oct 30 22:33:11.679564 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:11.679471 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:33:12.117862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.117826 5036 generic.go:356] "Generic (PLEG): container finished" podID="828f9550-05b1-486c-b332-fe3d7f4d701b" containerID="4c243c9748ad167c536fa9a5f593d9c747b68b5ab82dde25d5d998f7c6668424" exitCode=0 Oct 30 22:33:12.118299 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.117916 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" event={"ID":"828f9550-05b1-486c-b332-fe3d7f4d701b","Type":"ContainerDied","Data":"4c243c9748ad167c536fa9a5f593d9c747b68b5ab82dde25d5d998f7c6668424"} Oct 30 22:33:12.179030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.179001 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc"] Oct 30 22:33:12.183648 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.183591 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.203197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.203165 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858ec50c-3a05-49b3-ac34-0f74ddc80ab6" path="/var/lib/kubelet/pods/858ec50c-3a05-49b3-ac34-0f74ddc80ab6/volumes" Oct 30 22:33:12.203698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.203678 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc"] Oct 30 22:33:12.239041 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.239005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.239205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.239082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhh5\" (UniqueName: \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.239332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.239222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.341451 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.341412 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.341634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.341489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhh5\" (UniqueName: \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.341634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.341567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.342254 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.342230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.344067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.344034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.353141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.353116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhh5\" (UniqueName: \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\") pod \"maas-api-revoke-sub-0c00211a-9wqsc\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.510284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.510171 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:12.847405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:12.847371 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc"] Oct 30 22:33:12.848888 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:12.848860 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5717723_7c1e_4bde_995e_0021e5ed5fe0.slice/crio-99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7 WatchSource:0}: Error finding container 99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7: Status 404 returned error can't find the container with id 99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7 Oct 30 22:33:13.188801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.188760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" event={"ID":"e5717723-7c1e-4bde-995e-0021e5ed5fe0","Type":"ContainerStarted","Data":"99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7"} Oct 30 22:33:13.470403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.470377 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:13.608849 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.608816 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\" (UniqueName: \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\") pod \"828f9550-05b1-486c-b332-fe3d7f4d701b\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " Oct 30 22:33:13.609030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.609018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\") pod \"828f9550-05b1-486c-b332-fe3d7f4d701b\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " Oct 30 22:33:13.609097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.609058 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\") pod \"828f9550-05b1-486c-b332-fe3d7f4d701b\" (UID: \"828f9550-05b1-486c-b332-fe3d7f4d701b\") " Oct 30 22:33:13.609459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.609431 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca" pod "828f9550-05b1-486c-b332-fe3d7f4d701b" (UID: "828f9550-05b1-486c-b332-fe3d7f4d701b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:13.609608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.609592 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:13.611575 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.611548 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s" pod "828f9550-05b1-486c-b332-fe3d7f4d701b" (UID: "828f9550-05b1-486c-b332-fe3d7f4d701b"). InnerVolumeSpecName "kube-api-access-9pb7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:13.611687 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.611598 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca" pod "828f9550-05b1-486c-b332-fe3d7f4d701b" (UID: "828f9550-05b1-486c-b332-fe3d7f4d701b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:13.694183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.694151 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn"] Oct 30 22:33:13.694869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.694855 5036 cpu_manager.go:414] "containerMap: removing container" podUID="828f9550-05b1-486c-b332-fe3d7f4d701b" containerName="revoke-keys" Oct 30 22:33:13.694917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.694871 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828f9550-05b1-486c-b332-fe3d7f4d701b" containerName="revoke-keys" Oct 30 22:33:13.694994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.694984 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="828f9550-05b1-486c-b332-fe3d7f4d701b" containerName="revoke-keys" Oct 30 22:33:13.698816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.698762 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.710943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.710921 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/828f9550-05b1-486c-b332-fe3d7f4d701b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:13.711063 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.710944 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pb7s\" (UniqueName: \"kubernetes.io/projected/828f9550-05b1-486c-b332-fe3d7f4d701b-kube-api-access-9pb7s\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:13.717541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.717517 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn"] Oct 30 22:33:13.812527 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.812491 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.812684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.812602 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.812684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.812648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptnp\" (UniqueName: \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.914305 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.914247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.914469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.914383 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.914469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.914433 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptnp\" (UniqueName: \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.915020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.914985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.917088 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.917063 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:13.926590 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:13.926558 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptnp\" (UniqueName: \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:14.015207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.015134 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:14.201951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.201922 5036 generic.go:356] "Generic (PLEG): container finished" podID="e5717723-7c1e-4bde-995e-0021e5ed5fe0" containerID="e4e9ac630fa24a3a18c2f775976337c3cefbcd28461368e497a3c828742e3b55" exitCode=0 Oct 30 22:33:14.202417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.202054 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" event={"ID":"e5717723-7c1e-4bde-995e-0021e5ed5fe0","Type":"ContainerDied","Data":"e4e9ac630fa24a3a18c2f775976337c3cefbcd28461368e497a3c828742e3b55"} Oct 30 22:33:14.203637 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.203619 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" Oct 30 22:33:14.203770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.203663 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx" event={"ID":"828f9550-05b1-486c-b332-fe3d7f4d701b","Type":"ContainerDied","Data":"bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c"} Oct 30 22:33:14.203770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.203713 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff53ca50c08c2e7f00b0b81370b9299d86a811e63dd1833dc25d2f47469ba5c" Oct 30 22:33:14.329131 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:14.329100 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf389a6e_f9d8_4b9b_a5f5_84fcd60e5105.slice/crio-b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2 WatchSource:0}: Error finding container b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2: Status 404 returned error can't find the container with id b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2 Oct 30 22:33:14.329521 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:14.329489 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn"] Oct 30 22:33:15.032813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.032783 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:33:15.130983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.130897 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.130983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.130948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131033 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131062 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131157 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131300 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131345 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:15.131427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131442 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131486 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\") pod \"04a1df31-806c-4737-b116-2be2784a731b\" (UID: \"04a1df31-806c-4737-b116-2be2784a731b\") " Oct 30 22:33:15.131601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:15.131782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131670 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:15.131782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.131772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:15.132079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132055 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:15.132367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132301 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.132367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132331 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.132367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132347 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.132367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132362 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.132367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.132377 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a1df31-806c-4737-b116-2be2784a731b-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.133691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.133665 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:15.134189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.134158 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "kube-api-access-pslmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:15.134523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.134504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:15.134905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.134887 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy" pod "04a1df31-806c-4737-b116-2be2784a731b" (UID: "04a1df31-806c-4737-b116-2be2784a731b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:15.214143 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.214101 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" event={"ID":"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105","Type":"ContainerStarted","Data":"04fcd6ef9bac35a54353936661002d978bb50555c530f3f223f1f2516584fe61"} Oct 30 22:33:15.214638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.214163 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" event={"ID":"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105","Type":"ContainerStarted","Data":"b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2"} Oct 30 22:33:15.215812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.215786 5036 generic.go:356] "Generic (PLEG): container finished" podID="04a1df31-806c-4737-b116-2be2784a731b" containerID="7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5" exitCode=0 Oct 30 22:33:15.215936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.215838 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" Oct 30 22:33:15.215936 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.215872 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" event={"ID":"04a1df31-806c-4737-b116-2be2784a731b","Type":"ContainerDied","Data":"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5"} Oct 30 22:33:15.216023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.215961 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw" event={"ID":"04a1df31-806c-4737-b116-2be2784a731b","Type":"ContainerDied","Data":"0ca47b6b5a4982894a991772a0f2e48a16d03e706788780e8d3bcc1e3280ebcf"} Oct 30 22:33:15.216023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.215993 5036 scope.go:122] "RemoveContainer" containerID="7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5" Oct 30 22:33:15.232164 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.232145 5036 scope.go:122] "RemoveContainer" containerID="7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5" Oct 30 22:33:15.232558 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:15.232528 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5\": container with ID starting with 7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5 not found: ID does not exist" containerID="7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5" Oct 30 22:33:15.232690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.232564 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5"} err="failed to get container status \"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5\": rpc error: code = NotFound desc = could not find container \"7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5\": container with ID starting with 7acd09abe456b2af076ffe86a154098d750a15f84b2b0f66a57ad77c83e05af5 not found: ID does not exist" Oct 30 22:33:15.234855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.234169 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.234855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.234198 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a1df31-806c-4737-b116-2be2784a731b-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.234855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.234214 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslmk\" (UniqueName: \"kubernetes.io/projected/04a1df31-806c-4737-b116-2be2784a731b-kube-api-access-pslmk\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.234855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.234224 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a1df31-806c-4737-b116-2be2784a731b-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.240447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.240364 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" podStartSLOduration=2.240348384 podStartE2EDuration="2.240348384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:15.23377017 +0000 UTC m=+3219.183725108" watchObservedRunningTime="2026-10-30 22:33:15.240348384 +0000 UTC m=+3219.190303331" Oct 30 22:33:15.252924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.252894 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:33:15.259703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.259676 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-6nctw"] Oct 30 22:33:15.379242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.379215 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:15.435721 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.435684 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\") pod \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " Oct 30 22:33:15.436097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.436065 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\" (UniqueName: \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\") pod \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " Oct 30 22:33:15.436197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.436154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\") pod \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\" (UID: \"e5717723-7c1e-4bde-995e-0021e5ed5fe0\") " Oct 30 22:33:15.437431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.437400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca" pod "e5717723-7c1e-4bde-995e-0021e5ed5fe0" (UID: "e5717723-7c1e-4bde-995e-0021e5ed5fe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:15.439120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.439095 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca" pod "e5717723-7c1e-4bde-995e-0021e5ed5fe0" (UID: "e5717723-7c1e-4bde-995e-0021e5ed5fe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:15.439601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.439576 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5" pod "e5717723-7c1e-4bde-995e-0021e5ed5fe0" (UID: "e5717723-7c1e-4bde-995e-0021e5ed5fe0"). InnerVolumeSpecName "kube-api-access-fdhh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:15.538587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.538550 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhh5\" (UniqueName: \"kubernetes.io/projected/e5717723-7c1e-4bde-995e-0021e5ed5fe0-kube-api-access-fdhh5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.538587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.538588 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:15.538764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:15.538605 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5717723-7c1e-4bde-995e-0021e5ed5fe0-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:16.161606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.161573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:16.161794 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.161671 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:16.189745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.189716 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a1df31-806c-4737-b116-2be2784a731b" path="/var/lib/kubelet/pods/04a1df31-806c-4737-b116-2be2784a731b/volumes" Oct 30 22:33:16.230006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.229982 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" Oct 30 22:33:16.230431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.230004 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c00211a-9wqsc" event={"ID":"e5717723-7c1e-4bde-995e-0021e5ed5fe0","Type":"ContainerDied","Data":"99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7"} Oct 30 22:33:16.230431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.230059 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f09ac4deca49f8e96441c45bb65a752c73b79308a97d9640ab8603cbb7dfb7" Oct 30 22:33:16.232853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.232119 5036 generic.go:356] "Generic (PLEG): container finished" podID="bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" containerID="04fcd6ef9bac35a54353936661002d978bb50555c530f3f223f1f2516584fe61" exitCode=0 Oct 30 22:33:16.232853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:16.232214 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" event={"ID":"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105","Type":"ContainerDied","Data":"04fcd6ef9bac35a54353936661002d978bb50555c530f3f223f1f2516584fe61"} Oct 30 22:33:17.453742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.453712 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:17.580725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.580632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\" (UniqueName: \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\") pod \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " Oct 30 22:33:17.580901 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.580870 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\") pod \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " Oct 30 22:33:17.580973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.580929 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\") pod \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\" (UID: \"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105\") " Oct 30 22:33:17.581419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.581391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca" pod "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" (UID: "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:17.581537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.581518 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:17.583075 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.583052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp" pod "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" (UID: "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105"). InnerVolumeSpecName "kube-api-access-nptnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:17.583416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.583391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca" pod "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" (UID: "bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:17.682986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.682953 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nptnp\" (UniqueName: \"kubernetes.io/projected/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-kube-api-access-nptnp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:17.682986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:17.682983 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:18.100233 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:18.100192 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:33:18.258099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.258066 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" Oct 30 22:33:18.258099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.258075 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn" event={"ID":"bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105","Type":"ContainerDied","Data":"b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2"} Oct 30 22:33:18.258399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.258155 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5dec3013ee062a57fea9fb052ac54006ee3e79b5199e449907340288c5d5eb2" Oct 30 22:33:18.541554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.541519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:18.541554 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.541530 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:18.745854 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.745583 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 22:33:18.746402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.746101 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" containerID="cri-o://f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0" gracePeriod=30 Oct 30 22:33:18.887397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.886487 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:33:18.887397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:18.886567 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:33:19.615937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.615906 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 22:33:19.628609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.628584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\" (UniqueName: \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\") pod \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " Oct 30 22:33:19.628762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.628747 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\") pod \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\" (UID: \"15f5b8cf-7ce4-44d8-ba21-244a250c6740\") " Oct 30 22:33:19.629205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.629165 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume" pod "15f5b8cf-7ce4-44d8-ba21-244a250c6740" (UID: "15f5b8cf-7ce4-44d8-ba21-244a250c6740"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:19.631018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.630994 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs" pod "15f5b8cf-7ce4-44d8-ba21-244a250c6740" (UID: "15f5b8cf-7ce4-44d8-ba21-244a250c6740"). InnerVolumeSpecName "kube-api-access-6x4vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:19.667894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.667864 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 22:33:19.729933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.729899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\") pod \"b600fc49-9c31-48b8-bace-345729c4897e\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " Oct 30 22:33:19.730107 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.730040 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\" (UniqueName: \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\") pod \"b600fc49-9c31-48b8-bace-345729c4897e\" (UID: \"b600fc49-9c31-48b8-bace-345729c4897e\") " Oct 30 22:33:19.730416 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.730386 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume" pod "b600fc49-9c31-48b8-bace-345729c4897e" (UID: "b600fc49-9c31-48b8-bace-345729c4897e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:19.731000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.730872 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5b8cf-7ce4-44d8-ba21-244a250c6740-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:19.731102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.731032 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b600fc49-9c31-48b8-bace-345729c4897e-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:19.731102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.731051 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4vs\" (UniqueName: \"kubernetes.io/projected/15f5b8cf-7ce4-44d8-ba21-244a250c6740-kube-api-access-6x4vs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:19.733242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.733209 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj" pod "b600fc49-9c31-48b8-bace-345729c4897e" (UID: "b600fc49-9c31-48b8-bace-345729c4897e"). InnerVolumeSpecName "kube-api-access-gnnkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:19.832592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:19.832506 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnnkj\" (UniqueName: \"kubernetes.io/projected/b600fc49-9c31-48b8-bace-345729c4897e-kube-api-access-gnnkj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.278896 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.278858 5036 generic.go:356] "Generic (PLEG): container finished" podID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerID="62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534" exitCode=0 Oct 30 22:33:20.279057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.278904 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" Oct 30 22:33:20.279057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.278952 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" event={"ID":"15f5b8cf-7ce4-44d8-ba21-244a250c6740","Type":"ContainerDied","Data":"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534"} Oct 30 22:33:20.279057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.279009 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt" event={"ID":"15f5b8cf-7ce4-44d8-ba21-244a250c6740","Type":"ContainerDied","Data":"2c3b0eccfd54440a475ce218ae4f8b621f1177864086756827e8d428400aa8f0"} Oct 30 22:33:20.279057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.279036 5036 scope.go:122] "RemoveContainer" containerID="62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534" Oct 30 22:33:20.280622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.280589 5036 generic.go:356] "Generic (PLEG): container finished" podID="b600fc49-9c31-48b8-bace-345729c4897e" containerID="b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052" exitCode=0 Oct 30 22:33:20.280718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.280623 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" Oct 30 22:33:20.280785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.280718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" event={"ID":"b600fc49-9c31-48b8-bace-345729c4897e","Type":"ContainerDied","Data":"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052"} Oct 30 22:33:20.280785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.280764 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm" event={"ID":"b600fc49-9c31-48b8-bace-345729c4897e","Type":"ContainerDied","Data":"48bf3dfff662659cb87166bea4b6af8ad1f3dd905d6a45904e7c81cfa58c1543"} Oct 30 22:33:20.290873 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.290841 5036 scope.go:122] "RemoveContainer" containerID="62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534" Oct 30 22:33:20.291173 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:20.291142 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534\": container with ID starting with 62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534 not found: ID does not exist" containerID="62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534" Oct 30 22:33:20.291237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.291182 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534"} err="failed to get container status \"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534\": rpc error: code = NotFound desc = could not find container \"62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534\": container with ID starting with 62027cdc5f53bb63780f8159afc0116ec38863a88579e9651370f7785adc6534 not found: ID does not exist" Oct 30 22:33:20.291237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.291199 5036 scope.go:122] "RemoveContainer" containerID="b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052" Oct 30 22:33:20.301926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.301904 5036 scope.go:122] "RemoveContainer" containerID="b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052" Oct 30 22:33:20.302233 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:20.302214 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052\": container with ID starting with b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052 not found: ID does not exist" containerID="b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052" Oct 30 22:33:20.302302 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.302241 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052"} err="failed to get container status \"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052\": rpc error: code = NotFound desc = could not find container \"b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052\": container with ID starting with b1ce144299b901638e017f3ff9d4136902fefdf4faf2243a719fb85e33420052 not found: ID does not exist" Oct 30 22:33:20.305402 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.305383 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 22:33:20.311797 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.311774 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0f26c5-78b8676669-l8wpt"] Oct 30 22:33:20.323907 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.323886 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 22:33:20.333714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.333693 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0f26c5-759d959459-fq8wm"] Oct 30 22:33:20.357753 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:20.357731 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:33:22.189783 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:22.189747 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" path="/var/lib/kubelet/pods/15f5b8cf-7ce4-44d8-ba21-244a250c6740/volumes" Oct 30 22:33:22.190357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:22.190335 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b600fc49-9c31-48b8-bace-345729c4897e" path="/var/lib/kubelet/pods/b600fc49-9c31-48b8-bace-345729c4897e/volumes" Oct 30 22:33:22.769984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:22.769952 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:33:23.007189 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007153 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs"] Oct 30 22:33:23.007916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007893 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007917 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007955 5036 cpu_manager.go:414] "containerMap: removing container" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007963 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007988 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e5717723-7c1e-4bde-995e-0021e5ed5fe0" containerName="revoke-keys" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.007995 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5717723-7c1e-4bde-995e-0021e5ed5fe0" containerName="revoke-keys" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008022 5036 cpu_manager.go:414] "containerMap: removing container" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008029 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" Oct 30 22:33:23.008046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008043 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" containerName="revoke-keys" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008050 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" containerName="revoke-keys" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008210 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b600fc49-9c31-48b8-bace-345729c4897e" containerName="payload-pre-processing" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008227 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f5b8cf-7ce4-44d8-ba21-244a250c6740" containerName="payload-processing" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008240 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5717723-7c1e-4bde-995e-0021e5ed5fe0" containerName="revoke-keys" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008252 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a1df31-806c-4737-b116-2be2784a731b" containerName="istio-proxy" Oct 30 22:33:23.008508 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.008281 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" containerName="revoke-keys" Oct 30 22:33:23.022914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.022850 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs"] Oct 30 22:33:23.023050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.022948 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.026890 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.026861 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:23.028772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.028747 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:23.094079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.094050 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.094244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.094170 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8ql\" (UniqueName: \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.094244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.094226 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.102235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.102215 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:23.102335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.102315 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:23.195702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.195673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.196097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.195802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8ql\" (UniqueName: \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.196097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.195860 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.196455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.196428 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.198437 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.198414 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.207346 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.207327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8ql\" (UniqueName: \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\") pod \"maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.313681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.313643 5036 generic.go:356] "Generic (PLEG): container finished" podID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerID="5a4ebed3ba61c3846effb583552a2088d4ff45bf310f57c86a4a9490bd480018" exitCode=0 Oct 30 22:33:23.313880 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.313785 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" event={"ID":"ef5799f4-8428-4ea4-98bc-570c933a9b5a","Type":"ContainerDied","Data":"5a4ebed3ba61c3846effb583552a2088d4ff45bf310f57c86a4a9490bd480018"} Oct 30 22:33:23.315821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.315798 5036 generic.go:356] "Generic (PLEG): container finished" podID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerID="83c53512533b378cba3f2d8ead713de4fce173bb436012c82a3dfe7de9e7b934" exitCode=0 Oct 30 22:33:23.315940 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.315898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" event={"ID":"8519f1f5-1c5e-44e7-ad21-9f423750f1a7","Type":"ContainerDied","Data":"83c53512533b378cba3f2d8ead713de4fce173bb436012c82a3dfe7de9e7b934"} Oct 30 22:33:23.364065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.363570 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:23.474540 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.474517 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:33:23.596710 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.596686 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:33:23.611427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.611384 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\" (UniqueName: \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\") pod \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " Oct 30 22:33:23.611427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.611419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\") pod \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\" (UID: \"8519f1f5-1c5e-44e7-ad21-9f423750f1a7\") " Oct 30 22:33:23.611603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.611463 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\" (UniqueName: \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\") pod \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " Oct 30 22:33:23.612392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.612365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume" pod "8519f1f5-1c5e-44e7-ad21-9f423750f1a7" (UID: "8519f1f5-1c5e-44e7-ad21-9f423750f1a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:23.615911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.615171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg" pod "8519f1f5-1c5e-44e7-ad21-9f423750f1a7" (UID: "8519f1f5-1c5e-44e7-ad21-9f423750f1a7"). InnerVolumeSpecName "kube-api-access-msvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:23.615911 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.615235 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l" pod "ef5799f4-8428-4ea4-98bc-570c933a9b5a" (UID: "ef5799f4-8428-4ea4-98bc-570c933a9b5a"). InnerVolumeSpecName "kube-api-access-mc67l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:23.677083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.677010 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs"] Oct 30 22:33:23.677577 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:23.677551 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8478102_b86b_4c6a_b411_9bb9fae89cf6.slice/crio-b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe WatchSource:0}: Error finding container b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe: Status 404 returned error can't find the container with id b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe Oct 30 22:33:23.712745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.712718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\") pod \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\" (UID: \"ef5799f4-8428-4ea4-98bc-570c933a9b5a\") " Oct 30 22:33:23.713108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.713084 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume" pod "ef5799f4-8428-4ea4-98bc-570c933a9b5a" (UID: "ef5799f4-8428-4ea4-98bc-570c933a9b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:23.713394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.713376 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5799f4-8428-4ea4-98bc-570c933a9b5a-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:23.713445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.713402 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvwg\" (UniqueName: \"kubernetes.io/projected/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-kube-api-access-msvwg\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:23.713445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.713418 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519f1f5-1c5e-44e7-ad21-9f423750f1a7-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:23.713445 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.713432 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc67l\" (UniqueName: \"kubernetes.io/projected/ef5799f4-8428-4ea4-98bc-570c933a9b5a-kube-api-access-mc67l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:23.786412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:23.786386 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:33:24.260428 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.260403 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 22:33:24.330232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.330184 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" Oct 30 22:33:24.330232 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.330215 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9" event={"ID":"8519f1f5-1c5e-44e7-ad21-9f423750f1a7","Type":"ContainerDied","Data":"376cb2f39e69feedd39e1b99028ac811046e4d8e5b6880c85f204415a493ea20"} Oct 30 22:33:24.330560 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.330328 5036 scope.go:122] "RemoveContainer" containerID="83c53512533b378cba3f2d8ead713de4fce173bb436012c82a3dfe7de9e7b934" Oct 30 22:33:24.336764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.336038 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" event={"ID":"b8478102-b86b-4c6a-b411-9bb9fae89cf6","Type":"ContainerStarted","Data":"ab6ff5b5c373f82fdb53fe717c56ef44b261ac38520b08843f5240b06a5d3174"} Oct 30 22:33:24.336764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.336091 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" event={"ID":"b8478102-b86b-4c6a-b411-9bb9fae89cf6","Type":"ContainerStarted","Data":"b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe"} Oct 30 22:33:24.339861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.339841 5036 generic.go:356] "Generic (PLEG): container finished" podID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerID="f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0" exitCode=0 Oct 30 22:33:24.340054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.340036 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" event={"ID":"59612a38-c52a-48b7-9209-2e36fbc8007c","Type":"ContainerDied","Data":"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0"} Oct 30 22:33:24.340134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.340081 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" event={"ID":"59612a38-c52a-48b7-9209-2e36fbc8007c","Type":"ContainerDied","Data":"44feefa8a0dd919e1a02af6d00fd56abc94f2554e2369116b83d25081f517e26"} Oct 30 22:33:24.340207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.340174 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn" Oct 30 22:33:24.342256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.342235 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" event={"ID":"ef5799f4-8428-4ea4-98bc-570c933a9b5a","Type":"ContainerDied","Data":"62fe74992c2a6ccf7f1b349e89714ac47e39aaa4ad68932866fdf40cdd2ea205"} Oct 30 22:33:24.342440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.342424 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c" Oct 30 22:33:24.349946 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.349925 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.350057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.349969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.350057 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.350006 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.354000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.350101 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.354000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.350147 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.354000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.352324 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:33:24.354475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.354454 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:24.354871 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.354844 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:24.355224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.355194 5036 scope.go:122] "RemoveContainer" containerID="f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0" Oct 30 22:33:24.357711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.357681 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:24.358770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.358743 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecfbd6-5d5d58554f-gbwb9"] Oct 30 22:33:24.359435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.359410 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:24.359767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.359738 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:24.370192 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.370153 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" podStartSLOduration=2.370142183 podStartE2EDuration="2.370142183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:24.368651249 +0000 UTC m=+3228.318606197" watchObservedRunningTime="2026-10-30 22:33:24.370142183 +0000 UTC m=+3228.320097128" Oct 30 22:33:24.378412 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.378388 5036 scope.go:122] "RemoveContainer" containerID="f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0" Oct 30 22:33:24.378723 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:24.378705 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0\": container with ID starting with f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0 not found: ID does not exist" containerID="f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0" Oct 30 22:33:24.378787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.378731 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0"} err="failed to get container status \"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0\": rpc error: code = NotFound desc = could not find container \"f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0\": container with ID starting with f88a2ca25e2d1bdc1b4c6be5df9f37608c6734e98c0573520bcab29ceb9bffe0 not found: ID does not exist" Oct 30 22:33:24.378787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.378749 5036 scope.go:122] "RemoveContainer" containerID="5a4ebed3ba61c3846effb583552a2088d4ff45bf310f57c86a4a9490bd480018" Oct 30 22:33:24.384489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.384469 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:33:24.394123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.394101 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecfbd6-5b4cccff-l4s9c"] Oct 30 22:33:24.451467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451367 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.451467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.451726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451511 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.451726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451556 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\") pod \"59612a38-c52a-48b7-9209-2e36fbc8007c\" (UID: \"59612a38-c52a-48b7-9209-2e36fbc8007c\") " Oct 30 22:33:24.451726 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:24.451898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:24.451898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.451839 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:24.452227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452206 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452229 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59612a38-c52a-48b7-9209-2e36fbc8007c-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452241 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452251 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452260 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452288 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452297 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59612a38-c52a-48b7-9209-2e36fbc8007c-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.452319 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.452310 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.453958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.453930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774" pod "59612a38-c52a-48b7-9209-2e36fbc8007c" (UID: "59612a38-c52a-48b7-9209-2e36fbc8007c"). InnerVolumeSpecName "kube-api-access-7j774". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:24.553830 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.553796 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j774\" (UniqueName: \"kubernetes.io/projected/59612a38-c52a-48b7-9209-2e36fbc8007c-kube-api-access-7j774\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:24.673469 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.673430 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 22:33:24.681838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:24.681797 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-ctqpn"] Oct 30 22:33:25.357681 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.357647 5036 generic.go:356] "Generic (PLEG): container finished" podID="b8478102-b86b-4c6a-b411-9bb9fae89cf6" containerID="ab6ff5b5c373f82fdb53fe717c56ef44b261ac38520b08843f5240b06a5d3174" exitCode=0 Oct 30 22:33:25.358172 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.357732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" event={"ID":"b8478102-b86b-4c6a-b411-9bb9fae89cf6","Type":"ContainerDied","Data":"ab6ff5b5c373f82fdb53fe717c56ef44b261ac38520b08843f5240b06a5d3174"} Oct 30 22:33:25.358247 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.358218 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" podUID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" containerName="payload-processing" containerID="cri-o://c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59" gracePeriod=25 Oct 30 22:33:25.659616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.659590 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:33:25.679087 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.679052 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\" (UniqueName: \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\") pod \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " Oct 30 22:33:25.679562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.679543 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\") pod \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\" (UID: \"79f41ba0-9ecf-4a97-aa9c-7515e322cb72\") " Oct 30 22:33:25.680329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.680299 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume" pod "79f41ba0-9ecf-4a97-aa9c-7515e322cb72" (UID: "79f41ba0-9ecf-4a97-aa9c-7515e322cb72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:25.682879 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.682849 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k" pod "79f41ba0-9ecf-4a97-aa9c-7515e322cb72" (UID: "79f41ba0-9ecf-4a97-aa9c-7515e322cb72"). InnerVolumeSpecName "kube-api-access-bkx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:25.781706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.781615 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:25.781706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:25.781654 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkx2k\" (UniqueName: \"kubernetes.io/projected/79f41ba0-9ecf-4a97-aa9c-7515e322cb72-kube-api-access-bkx2k\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:26.189785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.189748 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" path="/var/lib/kubelet/pods/59612a38-c52a-48b7-9209-2e36fbc8007c/volumes" Oct 30 22:33:26.190312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.190293 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" path="/var/lib/kubelet/pods/8519f1f5-1c5e-44e7-ad21-9f423750f1a7/volumes" Oct 30 22:33:26.190782 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.190767 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" path="/var/lib/kubelet/pods/ef5799f4-8428-4ea4-98bc-570c933a9b5a/volumes" Oct 30 22:33:26.371159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.371124 5036 generic.go:356] "Generic (PLEG): container finished" podID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" containerID="c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59" exitCode=0 Oct 30 22:33:26.371624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.371173 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" Oct 30 22:33:26.371624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.371213 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" event={"ID":"79f41ba0-9ecf-4a97-aa9c-7515e322cb72","Type":"ContainerDied","Data":"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59"} Oct 30 22:33:26.371624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.371307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc" event={"ID":"79f41ba0-9ecf-4a97-aa9c-7515e322cb72","Type":"ContainerDied","Data":"0133d83aee7ac56b210cf861864e916fc783d8d3cdd7bdc99823579ac2d69d7e"} Oct 30 22:33:26.371624 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.371340 5036 scope.go:122] "RemoveContainer" containerID="c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59" Oct 30 22:33:26.384113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.384076 5036 scope.go:122] "RemoveContainer" containerID="c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59" Oct 30 22:33:26.384463 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:26.384433 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59\": container with ID starting with c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59 not found: ID does not exist" containerID="c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59" Oct 30 22:33:26.384566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.384474 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59"} err="failed to get container status \"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59\": rpc error: code = NotFound desc = could not find container \"c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59\": container with ID starting with c2d23864f9c5fbcafca112c7d0f9d60cc82e436441eef04edbccd3b5e2fa0a59 not found: ID does not exist" Oct 30 22:33:26.392500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.392430 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:33:26.396588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.396562 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cca1c7-6449dff8c5-ckltc"] Oct 30 22:33:26.563739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.563709 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:26.591244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.591211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\") pod \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " Oct 30 22:33:26.591444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.591295 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\" (UniqueName: \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\") pod \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " Oct 30 22:33:26.591515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.591498 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\") pod \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\" (UID: \"b8478102-b86b-4c6a-b411-9bb9fae89cf6\") " Oct 30 22:33:26.592231 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.592201 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca" pod "b8478102-b86b-4c6a-b411-9bb9fae89cf6" (UID: "b8478102-b86b-4c6a-b411-9bb9fae89cf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:26.594015 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.593987 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql" pod "b8478102-b86b-4c6a-b411-9bb9fae89cf6" (UID: "b8478102-b86b-4c6a-b411-9bb9fae89cf6"). InnerVolumeSpecName "kube-api-access-fk8ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:26.594899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.594874 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca" pod "b8478102-b86b-4c6a-b411-9bb9fae89cf6" (UID: "b8478102-b86b-4c6a-b411-9bb9fae89cf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:26.693011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.692977 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:26.693011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.693007 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8ql\" (UniqueName: \"kubernetes.io/projected/b8478102-b86b-4c6a-b411-9bb9fae89cf6-kube-api-access-fk8ql\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:26.693011 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.693017 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8478102-b86b-4c6a-b411-9bb9fae89cf6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:26.704705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.704666 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" podUID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" containerName="main" containerID="cri-o://07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552" gracePeriod=45 Oct 30 22:33:26.975307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.975279 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:33:26.997157 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997126 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997295 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997501 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997511 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\" (UniqueName: \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.997878 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.997673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\") pod \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\" (UID: \"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab\") " Oct 30 22:33:26.998307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.998286 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:26.999026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.998997 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:26.999223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:26.999202 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:27.000085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.000057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:27.000785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.000765 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:27.001405 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.001380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r" pod "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" (UID: "9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab"). InnerVolumeSpecName "kube-api-access-6gb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:27.099434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.099360 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.099434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.099386 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.099434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.099395 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.099434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.099411 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gb2r\" (UniqueName: \"kubernetes.io/projected/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-kube-api-access-6gb2r\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.099434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.099422 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.382203 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.382174 5036 generic.go:356] "Generic (PLEG): container finished" podID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" containerID="07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552" exitCode=0 Oct 30 22:33:27.382675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.382226 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" Oct 30 22:33:27.382675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.382255 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" event={"ID":"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab","Type":"ContainerDied","Data":"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552"} Oct 30 22:33:27.382675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.382315 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8" event={"ID":"9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab","Type":"ContainerDied","Data":"058642d5bf795cc79669feec4af5904ab6cbf83bf353290c9e60f500278e3e8f"} Oct 30 22:33:27.382675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.382338 5036 scope.go:122] "RemoveContainer" containerID="07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552" Oct 30 22:33:27.385897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.385876 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" Oct 30 22:33:27.386016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.385928 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs" event={"ID":"b8478102-b86b-4c6a-b411-9bb9fae89cf6","Type":"ContainerDied","Data":"b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe"} Oct 30 22:33:27.386016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.385985 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ecb5d3168f045d32b5074c7df1daed230eadc44562478becf175e971ca98fe" Oct 30 22:33:27.394452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.394432 5036 scope.go:122] "RemoveContainer" containerID="07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552" Oct 30 22:33:27.394762 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:27.394729 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552\": container with ID starting with 07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552 not found: ID does not exist" containerID="07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552" Oct 30 22:33:27.394853 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.394761 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552"} err="failed to get container status \"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552\": rpc error: code = NotFound desc = could not find container \"07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552\": container with ID starting with 07bff3702dad9d98c1acf3ef653d2b2483bb8913e781eecbf64a45875ccb2552 not found: ID does not exist" Oct 30 22:33:27.409606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.409583 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:33:27.421195 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:27.421172 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f60026/test-model-f60026-kserve-66b786576c-t68s8"] Oct 30 22:33:28.189840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:28.189801 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" path="/var/lib/kubelet/pods/79f41ba0-9ecf-4a97-aa9c-7515e322cb72/volumes" Oct 30 22:33:28.190284 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:28.190255 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" path="/var/lib/kubelet/pods/9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab/volumes" Oct 30 22:33:28.786696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:28.786650 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" containerID="cri-o://2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8" gracePeriod=25 Oct 30 22:33:30.393559 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:30.393520 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx"] Oct 30 22:33:30.397343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:30.397313 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cca1c7-h6zmx"] Oct 30 22:33:31.373924 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:31.373881 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn"] Oct 30 22:33:31.377470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:31.377445 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0a2577-7sbzn"] Oct 30 22:33:32.189139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:32.189102 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828f9550-05b1-486c-b332-fe3d7f4d701b" path="/var/lib/kubelet/pods/828f9550-05b1-486c-b332-fe3d7f4d701b/volumes" Oct 30 22:33:32.189639 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:32.189604 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105" path="/var/lib/kubelet/pods/bf389a6e-f9d8-4b9b-a5f5-84fcd60e5105/volumes" Oct 30 22:33:33.193596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:33.193569 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:33:33.213811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:33.213785 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:33:37.537925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:37.537889 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs"] Oct 30 22:33:37.545561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:37.545536 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f60026-7bdjs"] Oct 30 22:33:37.791626 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:37.791534 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" podUID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" containerName="main" containerID="cri-o://dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051" gracePeriod=45 Oct 30 22:33:38.201313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.194500 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" podUID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" containerName="payload-processing" containerID="cri-o://1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50" gracePeriod=25 Oct 30 22:33:38.202059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.202025 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8478102-b86b-4c6a-b411-9bb9fae89cf6" path="/var/lib/kubelet/pods/b8478102-b86b-4c6a-b411-9bb9fae89cf6/volumes" Oct 30 22:33:38.233255 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.215212 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" containerID="cri-o://5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab" gracePeriod=25 Oct 30 22:33:38.358120 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.358085 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx"] Oct 30 22:33:38.359215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359186 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" containerName="main" Oct 30 22:33:38.359400 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359380 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" containerName="main" Oct 30 22:33:38.359526 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359515 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" Oct 30 22:33:38.359610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359597 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" Oct 30 22:33:38.359712 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359701 5036 cpu_manager.go:414] "containerMap: removing container" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" Oct 30 22:33:38.359795 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359784 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" Oct 30 22:33:38.359887 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359877 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" Oct 30 22:33:38.359978 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.359967 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" Oct 30 22:33:38.360085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360076 5036 cpu_manager.go:414] "containerMap: removing container" podUID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" containerName="payload-processing" Oct 30 22:33:38.360166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360157 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" containerName="payload-processing" Oct 30 22:33:38.360306 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360295 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b8478102-b86b-4c6a-b411-9bb9fae89cf6" containerName="revoke-keys" Oct 30 22:33:38.360398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360387 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8478102-b86b-4c6a-b411-9bb9fae89cf6" containerName="revoke-keys" Oct 30 22:33:38.360661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360649 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8478102-b86b-4c6a-b411-9bb9fae89cf6" containerName="revoke-keys" Oct 30 22:33:38.360756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360747 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcb57ce-0a29-4fc8-8002-3dc9606fe0ab" containerName="main" Oct 30 22:33:38.360842 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360834 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5799f4-8428-4ea4-98bc-570c933a9b5a" containerName="payload-pre-processing" Oct 30 22:33:38.360922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360914 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="59612a38-c52a-48b7-9209-2e36fbc8007c" containerName="istio-proxy" Oct 30 22:33:38.360999 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.360991 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8519f1f5-1c5e-44e7-ad21-9f423750f1a7" containerName="payload-processing" Oct 30 22:33:38.361079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.361070 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f41ba0-9ecf-4a97-aa9c-7515e322cb72" containerName="payload-processing" Oct 30 22:33:38.380361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.379463 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.381691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.379342 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx"] Oct 30 22:33:38.383957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.383687 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:38.383957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.383750 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:38.383957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.383855 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:38.383957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.383918 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:38.414394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.414313 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449523 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449669 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\" (UniqueName: \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449894 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449942 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\" (UniqueName: \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.449982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\") pod \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\" (UID: \"91d02fba-c5ff-4bfd-a97f-95d3b5d48197\") " Oct 30 22:33:38.450384 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.450376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.450937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.450540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvcv\" (UniqueName: \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.450937 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.450650 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.453308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.451433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:38.453308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.451661 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:38.453308 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.452428 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:38.470950 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.470901 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "kube-api-access-5gnsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:38.480439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.473801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:38.484091 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.483756 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs" pod "91d02fba-c5ff-4bfd-a97f-95d3b5d48197" (UID: "91d02fba-c5ff-4bfd-a97f-95d3b5d48197"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:38.552197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552164 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.552673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.552673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvcv\" (UniqueName: \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.552812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552695 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.552812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552717 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gnsz\" (UniqueName: \"kubernetes.io/projected/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-kube-api-access-5gnsz\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.552923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552738 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.552923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552861 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.552923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552873 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.552923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.552886 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d02fba-c5ff-4bfd-a97f-95d3b5d48197-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.553375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.553352 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.554968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.554941 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.558347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.558328 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:33:38.564440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.564418 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvcv\" (UniqueName: \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.611749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.611715 5036 generic.go:356] "Generic (PLEG): container finished" podID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" containerID="1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50" exitCode=0 Oct 30 22:33:38.613129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.612877 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" event={"ID":"051b4d4f-ad8c-4583-828c-74ebe8aba86f","Type":"ContainerDied","Data":"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50"} Oct 30 22:33:38.613129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.612958 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" event={"ID":"051b4d4f-ad8c-4583-828c-74ebe8aba86f","Type":"ContainerDied","Data":"9321beb38916d395a21cbc20eae0c960e670075a70824dcb81843011f96c53aa"} Oct 30 22:33:38.613129 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.612996 5036 scope.go:122] "RemoveContainer" containerID="1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50" Oct 30 22:33:38.613696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.613463 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s" Oct 30 22:33:38.650499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.650464 5036 generic.go:356] "Generic (PLEG): container finished" podID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" containerID="dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051" exitCode=0 Oct 30 22:33:38.650699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.650557 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" Oct 30 22:33:38.650699 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.650688 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" event={"ID":"91d02fba-c5ff-4bfd-a97f-95d3b5d48197","Type":"ContainerDied","Data":"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051"} Oct 30 22:33:38.650825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.650732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27" event={"ID":"91d02fba-c5ff-4bfd-a97f-95d3b5d48197","Type":"ContainerDied","Data":"1c18ff8edad63139e1f04f7e6005674531c8d9210a59566fd5d262932484858f"} Oct 30 22:33:38.654781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.654753 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\") pod \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " Oct 30 22:33:38.655077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.655058 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\" (UniqueName: \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\") pod \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\" (UID: \"051b4d4f-ad8c-4583-828c-74ebe8aba86f\") " Oct 30 22:33:38.661309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.661283 5036 scope.go:122] "RemoveContainer" containerID="1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50" Oct 30 22:33:38.662412 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:38.662288 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50\": container with ID starting with 1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50 not found: ID does not exist" containerID="1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50" Oct 30 22:33:38.662654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.662422 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50"} err="failed to get container status \"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50\": rpc error: code = NotFound desc = could not find container \"1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50\": container with ID starting with 1a98d6f27612cb31be7fe1478db07c9489786c363b16f05b1ba3f82a1ac3af50 not found: ID does not exist" Oct 30 22:33:38.662736 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.662660 5036 scope.go:122] "RemoveContainer" containerID="dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051" Oct 30 22:33:38.663083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.663053 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume" pod "051b4d4f-ad8c-4583-828c-74ebe8aba86f" (UID: "051b4d4f-ad8c-4583-828c-74ebe8aba86f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:38.664858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.664801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22" pod "051b4d4f-ad8c-4583-828c-74ebe8aba86f" (UID: "051b4d4f-ad8c-4583-828c-74ebe8aba86f"). InnerVolumeSpecName "kube-api-access-sqt22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:38.688342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.688318 5036 scope.go:122] "RemoveContainer" containerID="dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051" Oct 30 22:33:38.688852 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:38.688821 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051\": container with ID starting with dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051 not found: ID does not exist" containerID="dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051" Oct 30 22:33:38.688957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.688883 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051"} err="failed to get container status \"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051\": rpc error: code = NotFound desc = could not find container \"dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051\": container with ID starting with dcd62f2102b9d24c1df4d363642c4dcd0570dea839a673cc642822e89970e051 not found: ID does not exist" Oct 30 22:33:38.690016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.689992 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:33:38.697528 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.697504 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-06ac2e/test-model-06ac2e-kserve-746797cb77-m6v27"] Oct 30 22:33:38.704584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.704558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:38.712601 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.712582 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:38.763175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.763081 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqt22\" (UniqueName: \"kubernetes.io/projected/051b4d4f-ad8c-4583-828c-74ebe8aba86f-kube-api-access-sqt22\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.763175 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.763113 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b4d4f-ad8c-4583-828c-74ebe8aba86f-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:38.940498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.940428 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:33:38.944365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:38.944337 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f60026-7bf67cfffc-2tl8s"] Oct 30 22:33:39.004973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.004944 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx"] Oct 30 22:33:39.006372 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:39.006347 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985f6fe1_e08c_4f3e_8849_7d5ed26195b5.slice/crio-0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf WatchSource:0}: Error finding container 0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf: Status 404 returned error can't find the container with id 0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf Oct 30 22:33:39.465647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.465608 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:33:39.465865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.465837 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" containerID="cri-o://9d8e960d4b1dd438b4c6b58c1485a7f9c51b61747dd72a5d4be570da10563387" gracePeriod=30 Oct 30 22:33:39.664657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.664618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" event={"ID":"985f6fe1-e08c-4f3e-8849-7d5ed26195b5","Type":"ContainerStarted","Data":"25b7f0bd6f8e13ba459d8a499f926ea4a3900fc9167e5608e977e362406c4c79"} Oct 30 22:33:39.665213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.664682 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" event={"ID":"985f6fe1-e08c-4f3e-8849-7d5ed26195b5","Type":"ContainerStarted","Data":"0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf"} Oct 30 22:33:39.685297 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:39.685224 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" podStartSLOduration=1.685211491 podStartE2EDuration="1.685211491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:39.681744812 +0000 UTC m=+3243.631699761" watchObservedRunningTime="2026-10-30 22:33:39.685211491 +0000 UTC m=+3243.635166434" Oct 30 22:33:40.189657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.189576 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" path="/var/lib/kubelet/pods/051b4d4f-ad8c-4583-828c-74ebe8aba86f/volumes" Oct 30 22:33:40.190208 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.190184 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" path="/var/lib/kubelet/pods/91d02fba-c5ff-4bfd-a97f-95d3b5d48197/volumes" Oct 30 22:33:40.206082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206057 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94"] Oct 30 22:33:40.206664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206650 5036 cpu_manager.go:414] "containerMap: removing container" podUID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" containerName="payload-processing" Oct 30 22:33:40.206767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206666 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" containerName="payload-processing" Oct 30 22:33:40.206767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206682 5036 cpu_manager.go:414] "containerMap: removing container" podUID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" containerName="main" Oct 30 22:33:40.206767 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206688 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" containerName="main" Oct 30 22:33:40.206881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206799 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="051b4d4f-ad8c-4583-828c-74ebe8aba86f" containerName="payload-processing" Oct 30 22:33:40.206881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.206809 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d02fba-c5ff-4bfd-a97f-95d3b5d48197" containerName="main" Oct 30 22:33:40.210525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.210505 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.234072 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.230091 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94"] Oct 30 22:33:40.286898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.286864 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.287062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.286928 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7rn\" (UniqueName: \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.287062 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.287033 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.388745 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.388695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7rn\" (UniqueName: \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.388916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.388789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.389000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.388984 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.389507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.389478 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.391558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.391539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.401176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.401150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7rn\" (UniqueName: \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\") pod \"maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.545335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.545233 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:40.679354 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.679310 5036 generic.go:356] "Generic (PLEG): container finished" podID="985f6fe1-e08c-4f3e-8849-7d5ed26195b5" containerID="25b7f0bd6f8e13ba459d8a499f926ea4a3900fc9167e5608e977e362406c4c79" exitCode=0 Oct 30 22:33:40.679812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.679377 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" event={"ID":"985f6fe1-e08c-4f3e-8849-7d5ed26195b5","Type":"ContainerDied","Data":"25b7f0bd6f8e13ba459d8a499f926ea4a3900fc9167e5608e977e362406c4c79"} Oct 30 22:33:40.842097 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:40.841248 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc46eaa_e5b4_4de3_8dc7_45280ac650ee.slice/crio-cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c WatchSource:0}: Error finding container cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c: Status 404 returned error can't find the container with id cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c Oct 30 22:33:40.843206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:40.843167 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94"] Oct 30 22:33:41.689102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.689073 5036 generic.go:356] "Generic (PLEG): container finished" podID="dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" containerID="c1fa38a1d41b90b75a2217885ca69002d3327cb93ea7959a7959c4258e95497a" exitCode=0 Oct 30 22:33:41.689541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.689160 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" event={"ID":"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee","Type":"ContainerDied","Data":"c1fa38a1d41b90b75a2217885ca69002d3327cb93ea7959a7959c4258e95497a"} Oct 30 22:33:41.689541 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.689221 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" event={"ID":"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee","Type":"ContainerStarted","Data":"cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c"} Oct 30 22:33:41.866909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.866883 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:41.942347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.941698 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\") pod \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " Oct 30 22:33:41.942347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.941973 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\" (UniqueName: \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\") pod \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " Oct 30 22:33:41.942347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.942029 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\") pod \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\" (UID: \"985f6fe1-e08c-4f3e-8849-7d5ed26195b5\") " Oct 30 22:33:41.942725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.942690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca" pod "985f6fe1-e08c-4f3e-8849-7d5ed26195b5" (UID: "985f6fe1-e08c-4f3e-8849-7d5ed26195b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:41.943483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.943460 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:41.956318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.945712 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca" pod "985f6fe1-e08c-4f3e-8849-7d5ed26195b5" (UID: "985f6fe1-e08c-4f3e-8849-7d5ed26195b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:41.956318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.952510 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:41.956318 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:41.953758 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv" pod "985f6fe1-e08c-4f3e-8849-7d5ed26195b5" (UID: "985f6fe1-e08c-4f3e-8849-7d5ed26195b5"). InnerVolumeSpecName "kube-api-access-btvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:42.044394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.044364 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvcv\" (UniqueName: \"kubernetes.io/projected/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-kube-api-access-btvcv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:42.044394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.044391 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985f6fe1-e08c-4f3e-8849-7d5ed26195b5-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:42.727914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.727877 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" Oct 30 22:33:42.728375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.727887 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx" event={"ID":"985f6fe1-e08c-4f3e-8849-7d5ed26195b5","Type":"ContainerDied","Data":"0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf"} Oct 30 22:33:42.728375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.728036 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5ee2eb4f2db349a55b9fdaaed59da3a080f407878b565928b9e8c2e4c041bf" Oct 30 22:33:42.905234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.905211 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:42.966968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.966928 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:33:42.967146 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:42.967004 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:33:43.058523 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.058431 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\") pod \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " Oct 30 22:33:43.058711 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.058690 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\" (UniqueName: \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\") pod \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " Oct 30 22:33:43.058798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.058744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\") pod \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\" (UID: \"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee\") " Oct 30 22:33:43.058867 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.058822 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca" pod "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" (UID: "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:43.059320 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.059292 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.061049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.061024 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn" pod "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" (UID: "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee"). InnerVolumeSpecName "kube-api-access-7f7rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:43.061165 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.061121 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca" pod "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" (UID: "dbc46eaa-e5b4-4de3-8dc7-45280ac650ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:43.160700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.160653 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f7rn\" (UniqueName: \"kubernetes.io/projected/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-kube-api-access-7f7rn\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.160700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.160697 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.737955 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.737924 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" Oct 30 22:33:43.738458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.737962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94" event={"ID":"dbc46eaa-e5b4-4de3-8dc7-45280ac650ee","Type":"ContainerDied","Data":"cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c"} Oct 30 22:33:43.738458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:43.738012 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad7e11d4418f3e75fee0a7aa0c7e64cbc6822f37f982e963da9cbe8cb3b4e3c" Oct 30 22:33:44.749038 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.749009 5036 generic.go:356] "Generic (PLEG): container finished" podID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerID="9d8e960d4b1dd438b4c6b58c1485a7f9c51b61747dd72a5d4be570da10563387" exitCode=0 Oct 30 22:33:44.749394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.749079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" event={"ID":"085c6a13-c475-4a63-a664-9d2bfdd584b5","Type":"ContainerDied","Data":"9d8e960d4b1dd438b4c6b58c1485a7f9c51b61747dd72a5d4be570da10563387"} Oct 30 22:33:44.774196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.774164 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:33:44.885770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885725 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.885990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.885990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.885990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885924 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.885990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.885990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.885990 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.886372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886022 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.886372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886093 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.886372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886281 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\") pod \"085c6a13-c475-4a63-a664-9d2bfdd584b5\" (UID: \"085c6a13-c475-4a63-a664-9d2bfdd584b5\") " Oct 30 22:33:44.886372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886326 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:44.886623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:44.886623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886558 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:44.886742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886694 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:44.886926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.886888 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:44.887205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.887183 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.887205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.887206 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.887367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.887216 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.887367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.887227 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085c6a13-c475-4a63-a664-9d2bfdd584b5-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.887367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.887235 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.888544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.888523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:44.888663 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.888646 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:44.889137 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.889120 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "kube-api-access-g5sl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:44.889501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.889483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy" pod "085c6a13-c475-4a63-a664-9d2bfdd584b5" (UID: "085c6a13-c475-4a63-a664-9d2bfdd584b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:44.969000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.968935 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:33:44.969220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.969191 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" containerID="cri-o://d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1" gracePeriod=30 Oct 30 22:33:44.988740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.988692 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.988740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.988725 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.988740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.988741 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5sl6\" (UniqueName: \"kubernetes.io/projected/085c6a13-c475-4a63-a664-9d2bfdd584b5-kube-api-access-g5sl6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:44.989055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:44.988758 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085c6a13-c475-4a63-a664-9d2bfdd584b5-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:45.466259 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.466222 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns"] Oct 30 22:33:45.467197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467172 5036 cpu_manager.go:414] "containerMap: removing container" podUID="985f6fe1-e08c-4f3e-8849-7d5ed26195b5" containerName="revoke-keys" Oct 30 22:33:45.467197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467199 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985f6fe1-e08c-4f3e-8849-7d5ed26195b5" containerName="revoke-keys" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467212 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" containerName="revoke-keys" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467221 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" containerName="revoke-keys" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467251 5036 cpu_manager.go:414] "containerMap: removing container" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467258 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467415 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="985f6fe1-e08c-4f3e-8849-7d5ed26195b5" containerName="revoke-keys" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467425 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" containerName="revoke-keys" Oct 30 22:33:45.467474 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.467441 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" containerName="istio-proxy" Oct 30 22:33:45.473223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.472901 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.477799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.476918 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:45.477799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.476942 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:45.485535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.485505 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns"] Oct 30 22:33:45.567883 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.567843 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:45.568050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.567955 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:45.599505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.599470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.599698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.599640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnszm\" (UniqueName: \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.599698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.599672 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.701260 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.701223 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.701481 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.701439 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.701580 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.701562 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnszm\" (UniqueName: \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.702117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.702056 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.703914 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.703893 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.713321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.713294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnszm\" (UniqueName: \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\") pod \"maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.763617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.762499 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" event={"ID":"085c6a13-c475-4a63-a664-9d2bfdd584b5","Type":"ContainerDied","Data":"537c068fa98f265ae4f4c81a8869708f12bd2e7d5d581a8790ff3e73b0767083"} Oct 30 22:33:45.763617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.762573 5036 scope.go:122] "RemoveContainer" containerID="9d8e960d4b1dd438b4c6b58c1485a7f9c51b61747dd72a5d4be570da10563387" Oct 30 22:33:45.763617 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.762729 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq" Oct 30 22:33:45.797908 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.795950 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:45.810343 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.810311 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:33:45.814173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:45.814152 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-rrsxq"] Oct 30 22:33:46.100490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:46.100463 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns"] Oct 30 22:33:46.102475 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:46.102436 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610964cf_0f54_42c9_bf15_ee89817322eb.slice/crio-b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f WatchSource:0}: Error finding container b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f: Status 404 returned error can't find the container with id b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f Oct 30 22:33:46.188610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:46.188577 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085c6a13-c475-4a63-a664-9d2bfdd584b5" path="/var/lib/kubelet/pods/085c6a13-c475-4a63-a664-9d2bfdd584b5/volumes" Oct 30 22:33:46.772610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:46.772512 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" event={"ID":"610964cf-0f54-42c9-bf15-ee89817322eb","Type":"ContainerStarted","Data":"1cae0484d4bccc71821fe19919defb6edb1bd27bc77111e7b0f583cd110c103c"} Oct 30 22:33:46.772610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:46.772597 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" event={"ID":"610964cf-0f54-42c9-bf15-ee89817322eb","Type":"ContainerStarted","Data":"b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f"} Oct 30 22:33:46.794994 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:46.794940 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" podStartSLOduration=1.794925846 podStartE2EDuration="1.794925846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:46.792755698 +0000 UTC m=+3250.742710658" watchObservedRunningTime="2026-10-30 22:33:46.794925846 +0000 UTC m=+3250.744880792" Oct 30 22:33:47.783698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:47.783664 5036 generic.go:356] "Generic (PLEG): container finished" podID="610964cf-0f54-42c9-bf15-ee89817322eb" containerID="1cae0484d4bccc71821fe19919defb6edb1bd27bc77111e7b0f583cd110c103c" exitCode=0 Oct 30 22:33:47.784098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:47.783753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" event={"ID":"610964cf-0f54-42c9-bf15-ee89817322eb","Type":"ContainerDied","Data":"1cae0484d4bccc71821fe19919defb6edb1bd27bc77111e7b0f583cd110c103c"} Oct 30 22:33:48.099975 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:48.099881 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:33:48.383657 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:48.383625 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:33:48.710792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:48.710684 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:48.985307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:48.983059 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:49.053855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.053819 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\") pod \"610964cf-0f54-42c9-bf15-ee89817322eb\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " Oct 30 22:33:49.054201 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.054179 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca" pod "610964cf-0f54-42c9-bf15-ee89817322eb" (UID: "610964cf-0f54-42c9-bf15-ee89817322eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:49.054628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.054604 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\") pod \"610964cf-0f54-42c9-bf15-ee89817322eb\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " Oct 30 22:33:49.054912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.054894 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\" (UniqueName: \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\") pod \"610964cf-0f54-42c9-bf15-ee89817322eb\" (UID: \"610964cf-0f54-42c9-bf15-ee89817322eb\") " Oct 30 22:33:49.055835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.055809 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:49.060056 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.059982 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca" pod "610964cf-0f54-42c9-bf15-ee89817322eb" (UID: "610964cf-0f54-42c9-bf15-ee89817322eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:49.060173 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.060135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm" pod "610964cf-0f54-42c9-bf15-ee89817322eb" (UID: "610964cf-0f54-42c9-bf15-ee89817322eb"). InnerVolumeSpecName "kube-api-access-pnszm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:49.091500 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.091474 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:33:49.157103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.157061 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnszm\" (UniqueName: \"kubernetes.io/projected/610964cf-0f54-42c9-bf15-ee89817322eb-kube-api-access-pnszm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:49.157389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.157110 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610964cf-0f54-42c9-bf15-ee89817322eb-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:49.250113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.250031 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:33:49.250113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.250093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:33:49.257582 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.257556 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\" (UniqueName: \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\") pod \"48e2bcd2-1cd7-4429-bef9-054536865311\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " Oct 30 22:33:49.257714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.257645 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\") pod \"48e2bcd2-1cd7-4429-bef9-054536865311\" (UID: \"48e2bcd2-1cd7-4429-bef9-054536865311\") " Oct 30 22:33:49.258017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.257983 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume" pod "48e2bcd2-1cd7-4429-bef9-054536865311" (UID: "48e2bcd2-1cd7-4429-bef9-054536865311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:49.258510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.258409 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e2bcd2-1cd7-4429-bef9-054536865311-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:49.260420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.260400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp" pod "48e2bcd2-1cd7-4429-bef9-054536865311" (UID: "48e2bcd2-1cd7-4429-bef9-054536865311"). InnerVolumeSpecName "kube-api-access-7zfmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:49.359394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.359361 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zfmp\" (UniqueName: \"kubernetes.io/projected/48e2bcd2-1cd7-4429-bef9-054536865311-kube-api-access-7zfmp\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.820363 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.820383 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns" event={"ID":"610964cf-0f54-42c9-bf15-ee89817322eb","Type":"ContainerDied","Data":"b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f"} Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.820448 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b08bb9b94f36ca21a005dce3a2070a991f9bd678043f3a8481e314b522cf473f" Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.822442 5036 generic.go:356] "Generic (PLEG): container finished" podID="48e2bcd2-1cd7-4429-bef9-054536865311" containerID="2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8" exitCode=0 Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.822489 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.822524 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" event={"ID":"48e2bcd2-1cd7-4429-bef9-054536865311","Type":"ContainerDied","Data":"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8"} Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.822579 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt" event={"ID":"48e2bcd2-1cd7-4429-bef9-054536865311","Type":"ContainerDied","Data":"ef762eefe35348412115f1f0f35506b9f24d8f8d68611d7733274f85805877a3"} Oct 30 22:33:49.830645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.822613 5036 scope.go:122] "RemoveContainer" containerID="2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8" Oct 30 22:33:49.843434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.843410 5036 scope.go:122] "RemoveContainer" containerID="2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8" Oct 30 22:33:49.844667 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:49.844637 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8\": container with ID starting with 2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8 not found: ID does not exist" containerID="2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8" Oct 30 22:33:49.844791 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.844673 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8"} err="failed to get container status \"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8\": rpc error: code = NotFound desc = could not find container \"2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8\": container with ID starting with 2544cf02d022b90fdbba1241a2fd1369d666544bb0f57dece25130eb1f0bf8b8 not found: ID does not exist" Oct 30 22:33:49.855811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.855786 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:33:49.861993 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:49.861966 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0a2577-7dbb797f75-c2jjt"] Oct 30 22:33:50.189605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.189570 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" path="/var/lib/kubelet/pods/48e2bcd2-1cd7-4429-bef9-054536865311/volumes" Oct 30 22:33:50.282616 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.282583 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:33:50.337810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.337766 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 22:33:50.339912 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.339881 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 22:33:50.371458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371386 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371437 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371495 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371591 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371698 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371635 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371702 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:50.371866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371720 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371804 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.371985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.371884 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\") pod \"17034993-a1cf-493a-a891-7e00d49f1a54\" (UID: \"17034993-a1cf-493a-a891-7e00d49f1a54\") " Oct 30 22:33:50.372410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.372380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:50.372558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.372462 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.372558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.372480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:50.372693 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.372662 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:50.372769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.372687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:50.374850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.374656 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:50.375122 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.375090 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:50.375398 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.375379 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:50.376367 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.376340 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs" pod "17034993-a1cf-493a-a891-7e00d49f1a54" (UID: "17034993-a1cf-493a-a891-7e00d49f1a54"). InnerVolumeSpecName "kube-api-access-bv9xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:50.474061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474022 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17034993-a1cf-493a-a891-7e00d49f1a54-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474056 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474071 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9xs\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-kube-api-access-bv9xs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474085 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474101 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17034993-a1cf-493a-a891-7e00d49f1a54-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474112 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474124 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17034993-a1cf-493a-a891-7e00d49f1a54-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.474301 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.474136 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17034993-a1cf-493a-a891-7e00d49f1a54-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:50.834819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.834779 5036 generic.go:356] "Generic (PLEG): container finished" podID="17034993-a1cf-493a-a891-7e00d49f1a54" containerID="d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1" exitCode=0 Oct 30 22:33:50.835001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.834911 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" Oct 30 22:33:50.835001 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.834919 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerDied","Data":"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1"} Oct 30 22:33:50.835151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.835015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t" event={"ID":"17034993-a1cf-493a-a891-7e00d49f1a54","Type":"ContainerDied","Data":"90ea8fa06f5315ad7ad82d828e585d9247242c08ba0e4e89449fb053fad76fad"} Oct 30 22:33:50.835151 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.835048 5036 scope.go:122] "RemoveContainer" containerID="d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1" Oct 30 22:33:50.851419 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.851395 5036 scope.go:122] "RemoveContainer" containerID="277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c" Oct 30 22:33:50.863843 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.863819 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:33:50.864728 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.864703 5036 scope.go:122] "RemoveContainer" containerID="d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1" Oct 30 22:33:50.865014 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:50.864991 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1\": container with ID starting with d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1 not found: ID does not exist" containerID="d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1" Oct 30 22:33:50.865154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.865026 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1"} err="failed to get container status \"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1\": rpc error: code = NotFound desc = could not find container \"d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1\": container with ID starting with d7d2dcfa78d1060b4ca5ec1f0cf6aeb695382642455ecbd356ca2733de15f6b1 not found: ID does not exist" Oct 30 22:33:50.865154 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.865052 5036 scope.go:122] "RemoveContainer" containerID="277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c" Oct 30 22:33:50.865422 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:50.865395 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c\": container with ID starting with 277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c not found: ID does not exist" containerID="277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c" Oct 30 22:33:50.865475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.865434 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c"} err="failed to get container status \"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c\": rpc error: code = NotFound desc = could not find container \"277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c\": container with ID starting with 277424c4fa219be597602e700cf23d9420c9fe0e9d6915d6c54ef532efb00e6c not found: ID does not exist" Oct 30 22:33:50.867643 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:50.867621 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-4vh9t"] Oct 30 22:33:51.016068 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.016035 5036 scope.go:122] "RemoveContainer" containerID="f1f70128b1cb7afa55a7971552243cd6906f70881ceea08f8da30dd895cb6378" Oct 30 22:33:51.027335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.027310 5036 scope.go:122] "RemoveContainer" containerID="e423c9aad3c149397dc41c0fcd9b5a23de66fbf929b1b609e130281659748077" Oct 30 22:33:51.042413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.042385 5036 scope.go:122] "RemoveContainer" containerID="24ec0181af1bc004821074b4d9cb88cd2063ce94cf350e10f1094483abcaddab" Oct 30 22:33:51.054689 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.054663 5036 scope.go:122] "RemoveContainer" containerID="9766521ddd7547fc975d9928b1b7ebc73dcc45d8df4f0f67d19b321517b2eb10" Oct 30 22:33:51.066219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.066198 5036 scope.go:122] "RemoveContainer" containerID="bb0454bd9cff5b310563d41613858e511f159871d6c99856d3fac3adbd564eb8" Oct 30 22:33:51.348609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.348521 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72"] Oct 30 22:33:51.349347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349323 5036 cpu_manager.go:414] "containerMap: removing container" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.349347 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349347 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.349598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349417 5036 cpu_manager.go:414] "containerMap: removing container" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" Oct 30 22:33:51.349598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349425 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" Oct 30 22:33:51.349598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349454 5036 cpu_manager.go:414] "containerMap: removing container" podUID="610964cf-0f54-42c9-bf15-ee89817322eb" containerName="revoke-keys" Oct 30 22:33:51.349598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349464 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610964cf-0f54-42c9-bf15-ee89817322eb" containerName="revoke-keys" Oct 30 22:33:51.349823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349617 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e2bcd2-1cd7-4429-bef9-054536865311" containerName="payload-processing" Oct 30 22:33:51.349823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349632 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.349823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349643 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="610964cf-0f54-42c9-bf15-ee89817322eb" containerName="revoke-keys" Oct 30 22:33:51.349823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.349655 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.357569 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.357541 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.359019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.358997 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv"] Oct 30 22:33:51.359823 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.359802 5036 cpu_manager.go:414] "containerMap: removing container" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.359927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.359824 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" containerName="istio-proxy" Oct 30 22:33:51.361489 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.361469 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:51.361668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.361649 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:51.361800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.361765 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:33:51.361923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.361899 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:51.365360 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.365338 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72"] Oct 30 22:33:51.365478 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.365424 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.388998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.388967 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv"] Oct 30 22:33:51.496544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496498 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.496775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496617 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.496775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.496775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496690 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgkj\" (UniqueName: \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.496949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496941 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.497013 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.496981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnsc\" (UniqueName: \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.598708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.598602 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.598708 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.598667 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.598964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.598710 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.598964 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.598741 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgkj\" (UniqueName: \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.599079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.598994 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.599079 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.599044 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnsc\" (UniqueName: \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.599496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.599454 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.599649 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.599456 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.601012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.600985 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx"] Oct 30 22:33:51.601709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.601686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.601820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.601741 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.607476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.607451 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0f73c9-dnffx"] Oct 30 22:33:51.611958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.611934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnsc\" (UniqueName: \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\") pod \"maas-api-revoke-sub-edadd22e-fqqwv\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.612067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.611964 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgkj\" (UniqueName: \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\") pod \"maas-api-revoke-sub-40106251-nmq72\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.687413 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.687373 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:51.774838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.774807 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:51.969638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.969583 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:51.992427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:51.992382 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72"] Oct 30 22:33:52.086134 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.086101 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv"] Oct 30 22:33:52.089863 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:33:52.089833 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939f7060_ddff_42d3_9423_cc23c7e8affd.slice/crio-bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628 WatchSource:0}: Error finding container bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628: Status 404 returned error can't find the container with id bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628 Oct 30 22:33:52.189502 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.189465 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17034993-a1cf-493a-a891-7e00d49f1a54" path="/var/lib/kubelet/pods/17034993-a1cf-493a-a891-7e00d49f1a54/volumes" Oct 30 22:33:52.190033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.190009 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985f6fe1-e08c-4f3e-8849-7d5ed26195b5" path="/var/lib/kubelet/pods/985f6fe1-e08c-4f3e-8849-7d5ed26195b5/volumes" Oct 30 22:33:52.859342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.859065 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" event={"ID":"939f7060-ddff-42d3-9423-cc23c7e8affd","Type":"ContainerStarted","Data":"1ec6cd614b1f3593dfa32187eccf93f385885edf59f2da6d17a5eff505e4460b"} Oct 30 22:33:52.859342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.859127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" event={"ID":"939f7060-ddff-42d3-9423-cc23c7e8affd","Type":"ContainerStarted","Data":"bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628"} Oct 30 22:33:52.861054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.861031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" event={"ID":"8f7d47fb-c7da-4768-9e7e-544369f5fc5f","Type":"ContainerStarted","Data":"8e45bec05133063a308718d26920cc38e53005d9cc26736cb799d3ffeffedcf5"} Oct 30 22:33:52.861178 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.861070 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" event={"ID":"8f7d47fb-c7da-4768-9e7e-544369f5fc5f","Type":"ContainerStarted","Data":"dea2d7f7bc0fb17cc6463e48b814e4163e87bd43c966a169f1ef6cf03b454b69"} Oct 30 22:33:52.879116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.879069 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" podStartSLOduration=1.879056965 podStartE2EDuration="1.879056965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:52.875793534 +0000 UTC m=+3256.825748491" watchObservedRunningTime="2026-10-30 22:33:52.879056965 +0000 UTC m=+3256.829011910" Oct 30 22:33:52.896186 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:52.896130 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" podStartSLOduration=1.896116545 podStartE2EDuration="1.896116545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:52.891995278 +0000 UTC m=+3256.841950223" watchObservedRunningTime="2026-10-30 22:33:52.896116545 +0000 UTC m=+3256.846071492" Oct 30 22:33:53.384729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:53.384685 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" containerID="cri-o://6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821" gracePeriod=25 Oct 30 22:33:53.870484 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:53.870394 5036 generic.go:356] "Generic (PLEG): container finished" podID="939f7060-ddff-42d3-9423-cc23c7e8affd" containerID="1ec6cd614b1f3593dfa32187eccf93f385885edf59f2da6d17a5eff505e4460b" exitCode=0 Oct 30 22:33:53.870895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:53.870486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" event={"ID":"939f7060-ddff-42d3-9423-cc23c7e8affd","Type":"ContainerDied","Data":"1ec6cd614b1f3593dfa32187eccf93f385885edf59f2da6d17a5eff505e4460b"} Oct 30 22:33:53.872237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:53.872208 5036 generic.go:356] "Generic (PLEG): container finished" podID="8f7d47fb-c7da-4768-9e7e-544369f5fc5f" containerID="8e45bec05133063a308718d26920cc38e53005d9cc26736cb799d3ffeffedcf5" exitCode=0 Oct 30 22:33:53.872397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:53.872239 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" event={"ID":"8f7d47fb-c7da-4768-9e7e-544369f5fc5f","Type":"ContainerDied","Data":"8e45bec05133063a308718d26920cc38e53005d9cc26736cb799d3ffeffedcf5"} Oct 30 22:33:55.062426 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.062393 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:55.096307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.096254 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:55.180787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.180709 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\") pod \"939f7060-ddff-42d3-9423-cc23c7e8affd\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " Oct 30 22:33:55.180928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.180825 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\") pod \"939f7060-ddff-42d3-9423-cc23c7e8affd\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " Oct 30 22:33:55.180928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.180879 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\" (UniqueName: \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\") pod \"939f7060-ddff-42d3-9423-cc23c7e8affd\" (UID: \"939f7060-ddff-42d3-9423-cc23c7e8affd\") " Oct 30 22:33:55.181239 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.181215 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca" pod "939f7060-ddff-42d3-9423-cc23c7e8affd" (UID: "939f7060-ddff-42d3-9423-cc23c7e8affd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:55.181596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.181526 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.183117 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.183091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca" pod "939f7060-ddff-42d3-9423-cc23c7e8affd" (UID: "939f7060-ddff-42d3-9423-cc23c7e8affd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:55.183290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.183248 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc" pod "939f7060-ddff-42d3-9423-cc23c7e8affd" (UID: "939f7060-ddff-42d3-9423-cc23c7e8affd"). InnerVolumeSpecName "kube-api-access-hbnsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:55.282498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.282472 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\" (UniqueName: \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\") pod \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " Oct 30 22:33:55.282673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.282648 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\") pod \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " Oct 30 22:33:55.282738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.282706 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\") pod \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\" (UID: \"8f7d47fb-c7da-4768-9e7e-544369f5fc5f\") " Oct 30 22:33:55.283105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.283076 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca" pod "8f7d47fb-c7da-4768-9e7e-544369f5fc5f" (UID: "8f7d47fb-c7da-4768-9e7e-544369f5fc5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:55.283505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.283242 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939f7060-ddff-42d3-9423-cc23c7e8affd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.283505 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.283282 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnsc\" (UniqueName: \"kubernetes.io/projected/939f7060-ddff-42d3-9423-cc23c7e8affd-kube-api-access-hbnsc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.284910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.284891 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj" pod "8f7d47fb-c7da-4768-9e7e-544369f5fc5f" (UID: "8f7d47fb-c7da-4768-9e7e-544369f5fc5f"). InnerVolumeSpecName "kube-api-access-9fgkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:55.285494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.285476 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca" pod "8f7d47fb-c7da-4768-9e7e-544369f5fc5f" (UID: "8f7d47fb-c7da-4768-9e7e-544369f5fc5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:55.339099 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.339033 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" containerID="cri-o://1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9" gracePeriod=25 Oct 30 22:33:55.346824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.346785 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" containerID="cri-o://e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02" gracePeriod=25 Oct 30 22:33:55.384171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.384141 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.384171 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.384170 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgkj\" (UniqueName: \"kubernetes.io/projected/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-kube-api-access-9fgkj\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.384380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.384181 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7d47fb-c7da-4768-9e7e-544369f5fc5f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:55.636608 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.636574 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:55.649166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.649140 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:33:55.665555 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.665524 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:33:55.891877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.891736 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" Oct 30 22:33:55.891877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.891742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edadd22e-fqqwv" event={"ID":"939f7060-ddff-42d3-9423-cc23c7e8affd","Type":"ContainerDied","Data":"bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628"} Oct 30 22:33:55.891877 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.891858 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf332ef97346e9eb52292e298a51c5cb4f3050f3015fad96e740c784fb527628" Oct 30 22:33:55.893553 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.893522 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" Oct 30 22:33:55.893684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.893584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40106251-nmq72" event={"ID":"8f7d47fb-c7da-4768-9e7e-544369f5fc5f","Type":"ContainerDied","Data":"dea2d7f7bc0fb17cc6463e48b814e4163e87bd43c966a169f1ef6cf03b454b69"} Oct 30 22:33:55.893684 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:55.893625 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea2d7f7bc0fb17cc6463e48b814e4163e87bd43c966a169f1ef6cf03b454b69" Oct 30 22:33:58.434032 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.434007 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:33:58.538934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.538846 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\" (UniqueName: \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\") pod \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " Oct 30 22:33:58.539118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.538957 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\") pod \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\" (UID: \"46c385c1-90d0-4da7-9c0a-012fa8fee7fd\") " Oct 30 22:33:58.539366 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.539343 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume" pod "46c385c1-90d0-4da7-9c0a-012fa8fee7fd" (UID: "46c385c1-90d0-4da7-9c0a-012fa8fee7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:58.539498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.539477 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:58.541160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.541137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv" pod "46c385c1-90d0-4da7-9c0a-012fa8fee7fd" (UID: "46c385c1-90d0-4da7-9c0a-012fa8fee7fd"). InnerVolumeSpecName "kube-api-access-rqvpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:58.640511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.640481 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvpv\" (UniqueName: \"kubernetes.io/projected/46c385c1-90d0-4da7-9c0a-012fa8fee7fd-kube-api-access-rqvpv\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:58.925070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.925036 5036 generic.go:356] "Generic (PLEG): container finished" podID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerID="5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab" exitCode=0 Oct 30 22:33:58.925335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.925090 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" event={"ID":"46c385c1-90d0-4da7-9c0a-012fa8fee7fd","Type":"ContainerDied","Data":"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab"} Oct 30 22:33:58.925335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.925104 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" Oct 30 22:33:58.925335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.925161 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8" event={"ID":"46c385c1-90d0-4da7-9c0a-012fa8fee7fd","Type":"ContainerDied","Data":"652c7f91b7b18028d4f03ee775fa09a4bead33a2137cf9d1d5e69be021ee80bf"} Oct 30 22:33:58.925335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.925188 5036 scope.go:122] "RemoveContainer" containerID="5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab" Oct 30 22:33:58.939339 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.939312 5036 scope.go:122] "RemoveContainer" containerID="5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab" Oct 30 22:33:58.939609 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:33:58.939583 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab\": container with ID starting with 5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab not found: ID does not exist" containerID="5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab" Oct 30 22:33:58.939709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.939621 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab"} err="failed to get container status \"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab\": rpc error: code = NotFound desc = could not find container \"5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab\": container with ID starting with 5c2bab1a6d13934bee6111669f1ff630633422dceb557a519ce1271143c95dab not found: ID does not exist" Oct 30 22:33:58.950435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.950409 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:33:58.956148 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:58.956125 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f60026-578b476b55-v95x8"] Oct 30 22:33:59.435439 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:59.435388 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:33:59.436066 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:59.436034 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" containerID="cri-o://59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b" gracePeriod=30 Oct 30 22:33:59.623340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:59.621321 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns"] Oct 30 22:33:59.626826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:33:59.626786 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-06ac2e-7lbns"] Oct 30 22:34:00.089739 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.089708 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4"] Oct 30 22:34:00.090334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090316 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8f7d47fb-c7da-4768-9e7e-544369f5fc5f" containerName="revoke-keys" Oct 30 22:34:00.090334 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090333 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7d47fb-c7da-4768-9e7e-544369f5fc5f" containerName="revoke-keys" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090351 5036 cpu_manager.go:414] "containerMap: removing container" podUID="939f7060-ddff-42d3-9423-cc23c7e8affd" containerName="revoke-keys" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090357 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939f7060-ddff-42d3-9423-cc23c7e8affd" containerName="revoke-keys" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090364 5036 cpu_manager.go:414] "containerMap: removing container" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090369 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090486 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7d47fb-c7da-4768-9e7e-544369f5fc5f" containerName="revoke-keys" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090498 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="939f7060-ddff-42d3-9423-cc23c7e8affd" containerName="revoke-keys" Oct 30 22:34:00.090524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.090503 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" containerName="payload-pre-processing" Oct 30 22:34:00.095356 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.095338 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.099357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.099335 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:00.099476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.099430 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:34:00.101434 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.101416 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:00.101562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.101460 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:00.102986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.102960 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4"] Oct 30 22:34:00.162535 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.162509 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.162692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.162553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.162759 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.162737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tt6\" (UniqueName: \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.188228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.188204 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c385c1-90d0-4da7-9c0a-012fa8fee7fd" path="/var/lib/kubelet/pods/46c385c1-90d0-4da7-9c0a-012fa8fee7fd/volumes" Oct 30 22:34:00.188656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.188641 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610964cf-0f54-42c9-bf15-ee89817322eb" path="/var/lib/kubelet/pods/610964cf-0f54-42c9-bf15-ee89817322eb/volumes" Oct 30 22:34:00.264303 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.264235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tt6\" (UniqueName: \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.264588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.264564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.264700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.264623 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.265332 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.265309 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.267166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.267147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.276645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.276618 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tt6\" (UniqueName: \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\") pod \"maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.411696 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.411661 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:00.649927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.649872 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" podUID="63377bef-457a-4395-9003-0c799b7c031b" containerName="payload-processing" containerID="cri-o://70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939" gracePeriod=25 Oct 30 22:34:00.666362 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.666252 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" containerID="cri-o://21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68" gracePeriod=25 Oct 30 22:34:00.714019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.713993 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4"] Oct 30 22:34:00.723530 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:34:00.723501 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f4dd6e_747b_45d1_8b31_7caa99b56d8e.slice/crio-c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4 WatchSource:0}: Error finding container c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4: Status 404 returned error can't find the container with id c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4 Oct 30 22:34:00.926781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.926714 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:34:00.953618 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.953579 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" event={"ID":"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e","Type":"ContainerStarted","Data":"c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4"} Oct 30 22:34:00.955462 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.955432 5036 generic.go:356] "Generic (PLEG): container finished" podID="63377bef-457a-4395-9003-0c799b7c031b" containerID="70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939" exitCode=0 Oct 30 22:34:00.955622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.955474 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" Oct 30 22:34:00.955622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.955524 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" event={"ID":"63377bef-457a-4395-9003-0c799b7c031b","Type":"ContainerDied","Data":"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939"} Oct 30 22:34:00.955622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.955585 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd" event={"ID":"63377bef-457a-4395-9003-0c799b7c031b","Type":"ContainerDied","Data":"d8e01ae84aa66fb8482f5f985dd81f805d8492bcd9d8d9aa4ce72d8195b1998a"} Oct 30 22:34:00.955622 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.955615 5036 scope.go:122] "RemoveContainer" containerID="70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939" Oct 30 22:34:00.969779 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.969757 5036 scope.go:122] "RemoveContainer" containerID="70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939" Oct 30 22:34:00.970120 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:00.970100 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939\": container with ID starting with 70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939 not found: ID does not exist" containerID="70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939" Oct 30 22:34:00.970182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.970132 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939"} err="failed to get container status \"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939\": rpc error: code = NotFound desc = could not find container \"70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939\": container with ID starting with 70432c0b088c5a9e92947359d1d4d95bf2b2f343a88e57fd16b89188ab838939 not found: ID does not exist" Oct 30 22:34:00.974661 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.974638 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\" (UniqueName: \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\") pod \"63377bef-457a-4395-9003-0c799b7c031b\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " Oct 30 22:34:00.974771 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.974681 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\") pod \"63377bef-457a-4395-9003-0c799b7c031b\" (UID: \"63377bef-457a-4395-9003-0c799b7c031b\") " Oct 30 22:34:00.975084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:00.975060 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume" pod "63377bef-457a-4395-9003-0c799b7c031b" (UID: "63377bef-457a-4395-9003-0c799b7c031b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:01.010973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.010951 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9" pod "63377bef-457a-4395-9003-0c799b7c031b" (UID: "63377bef-457a-4395-9003-0c799b7c031b"). InnerVolumeSpecName "kube-api-access-jl7t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:01.076533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.076506 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7t9\" (UniqueName: \"kubernetes.io/projected/63377bef-457a-4395-9003-0c799b7c031b-kube-api-access-jl7t9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:01.076533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.076530 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63377bef-457a-4395-9003-0c799b7c031b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:01.279956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.279930 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:34:01.286387 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.286361 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06ac2e-78cc6d9985-cxlxd"] Oct 30 22:34:01.967210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.967176 5036 generic.go:356] "Generic (PLEG): container finished" podID="e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" containerID="fdebe158f4c908dfda9be5a1cebc09444409b2003739915c1c2de77a22dca8b7" exitCode=0 Oct 30 22:34:01.967674 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:01.967295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" event={"ID":"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e","Type":"ContainerDied","Data":"fdebe158f4c908dfda9be5a1cebc09444409b2003739915c1c2de77a22dca8b7"} Oct 30 22:34:02.189520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:02.189488 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63377bef-457a-4395-9003-0c799b7c031b" path="/var/lib/kubelet/pods/63377bef-457a-4395-9003-0c799b7c031b/volumes" Oct 30 22:34:03.113352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.110592 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94"] Oct 30 22:34:03.118357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.118330 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1529dc-fwt94"] Oct 30 22:34:03.267113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.266410 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:03.413388 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.413351 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\" (UniqueName: \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\") pod \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " Oct 30 22:34:03.413816 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.413796 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\") pod \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " Oct 30 22:34:03.414059 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.414043 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\") pod \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\" (UID: \"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e\") " Oct 30 22:34:03.414706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.414584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca" pod "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" (UID: "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:03.415427 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.415397 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.417407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.417380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6" pod "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" (UID: "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e"). InnerVolumeSpecName "kube-api-access-x5tt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:03.417690 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.417672 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca" pod "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" (UID: "e5f4dd6e-747b-45d1-8b31-7caa99b56d8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:03.516279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.516215 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5tt6\" (UniqueName: \"kubernetes.io/projected/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-kube-api-access-x5tt6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.516279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:03.516246 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.024578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.024546 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" Oct 30 22:34:04.024578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.024570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4" event={"ID":"e5f4dd6e-747b-45d1-8b31-7caa99b56d8e","Type":"ContainerDied","Data":"c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4"} Oct 30 22:34:04.024861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.024643 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3423cce180947b6405399d7dbdc68628f6ddacb1239a26127b723f1618c2eb4" Oct 30 22:34:04.190235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.190201 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc46eaa-e5b4-4de3-8dc7-45280ac650ee" path="/var/lib/kubelet/pods/dbc46eaa-e5b4-4de3-8dc7-45280ac650ee/volumes" Oct 30 22:34:04.499487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.499450 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:04.499487 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.499474 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:04.733599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.733565 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:34:04.834123 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834156 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834317 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834362 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834424 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834534 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834572 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\") pod \"06cd4692-c724-4687-b9cc-066b9c2c04d5\" (UID: \"06cd4692-c724-4687-b9cc-066b9c2c04d5\") " Oct 30 22:34:04.834868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.834868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834606 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.834868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834719 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:04.834868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.835102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.834932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.835207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.835171 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cd4692-c724-4687-b9cc-066b9c2c04d5-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.835207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.835198 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.835368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.835214 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.835368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.835228 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.835368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.835243 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.836865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.836828 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:04.837160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.837141 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "kube-api-access-hkrt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:04.837406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.837387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.837471 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.837428 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token" pod "06cd4692-c724-4687-b9cc-066b9c2c04d5" (UID: "06cd4692-c724-4687-b9cc-066b9c2c04d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:04.936225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.936185 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.936225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.936222 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.936507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.936239 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrt9\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-kube-api-access-hkrt9\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.936507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.936254 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cd4692-c724-4687-b9cc-066b9c2c04d5-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.937206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:04.937174 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 22:34:05.034790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.034698 5036 generic.go:356] "Generic (PLEG): container finished" podID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerID="59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b" exitCode=0 Oct 30 22:34:05.034790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.034747 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerDied","Data":"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b"} Oct 30 22:34:05.034790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.034772 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" Oct 30 22:34:05.035050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.034796 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns" event={"ID":"06cd4692-c724-4687-b9cc-066b9c2c04d5","Type":"ContainerDied","Data":"e3b8ab92627ebac496d1d34c763c683d22fa7cf718a14772ad9584ab9c36c4ea"} Oct 30 22:34:05.035050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.034820 5036 scope.go:122] "RemoveContainer" containerID="59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b" Oct 30 22:34:05.079531 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.079377 5036 scope.go:122] "RemoveContainer" containerID="aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7" Oct 30 22:34:05.088358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.088327 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:34:05.092246 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.092217 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0f73c9-openshift-default-6879bd4895-gr7ns"] Oct 30 22:34:05.098902 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.098881 5036 scope.go:122] "RemoveContainer" containerID="59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b" Oct 30 22:34:05.099205 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:05.099184 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b\": container with ID starting with 59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b not found: ID does not exist" containerID="59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b" Oct 30 22:34:05.099335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.099213 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b"} err="failed to get container status \"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b\": rpc error: code = NotFound desc = could not find container \"59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b\": container with ID starting with 59845c63e96f0ae6d1738ab5ecae100ec637a8f9ff9013a0f16d0077a6464b8b not found: ID does not exist" Oct 30 22:34:05.099335 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.099242 5036 scope.go:122] "RemoveContainer" containerID="aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7" Oct 30 22:34:05.099599 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:05.099575 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7\": container with ID starting with aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7 not found: ID does not exist" containerID="aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7" Oct 30 22:34:05.099647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.099606 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7"} err="failed to get container status \"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7\": rpc error: code = NotFound desc = could not find container \"aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7\": container with ID starting with aa493c00e2dc10a8e9a487db93d4599163c59a6b39badea3c397ad456936dfb7 not found: ID does not exist" Oct 30 22:34:05.636307 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:05.636249 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:06.189760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:06.189719 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" path="/var/lib/kubelet/pods/06cd4692-c724-4687-b9cc-066b9c2c04d5/volumes" Oct 30 22:34:07.501466 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:07.501426 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:07.574288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:07.561756 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:34:07.574288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:07.562003 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" containerID="cri-o://dff8110a3d121f5d4c0de089da739589cfe537546895ca4ab2accd7cdc3c4fc1" gracePeriod=30 Oct 30 22:34:08.074394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074361 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f"] Oct 30 22:34:08.074913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074900 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" containerName="revoke-keys" Oct 30 22:34:08.074975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074915 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" containerName="revoke-keys" Oct 30 22:34:08.074975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074929 5036 cpu_manager.go:414] "containerMap: removing container" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:08.074975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074934 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:08.074975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074945 5036 cpu_manager.go:414] "containerMap: removing container" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:08.074975 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074951 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:08.075169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074982 5036 cpu_manager.go:414] "containerMap: removing container" podUID="63377bef-457a-4395-9003-0c799b7c031b" containerName="payload-processing" Oct 30 22:34:08.075169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.074987 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63377bef-457a-4395-9003-0c799b7c031b" containerName="payload-processing" Oct 30 22:34:08.075169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.075070 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:08.075169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.075084 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="63377bef-457a-4395-9003-0c799b7c031b" containerName="payload-processing" Oct 30 22:34:08.075169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.075093 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" containerName="revoke-keys" Oct 30 22:34:08.078885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.078863 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.084085 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.084002 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:08.084225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.084096 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:08.084423 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.084401 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:08.084543 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.084498 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:34:08.088949 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.088918 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f"] Oct 30 22:34:08.178738 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.178698 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gvm\" (UniqueName: \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.178917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.178776 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.178917 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.178857 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.280351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.280315 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gvm\" (UniqueName: \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.280534 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.280403 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.280598 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.280565 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.281127 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.281101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.283100 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.283081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.291819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.291800 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gvm\" (UniqueName: \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.398341 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.398313 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:08.686985 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:08.686955 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f"] Oct 30 22:34:08.688366 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:34:08.688338 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d2f934_75ce_4f00_b94d_5ac07009ecd2.slice/crio-6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5 WatchSource:0}: Error finding container 6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5: Status 404 returned error can't find the container with id 6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5 Oct 30 22:34:09.080111 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:09.080071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" event={"ID":"78d2f934-75ce-4f00-b94d-5ac07009ecd2","Type":"ContainerStarted","Data":"6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5"} Oct 30 22:34:10.090019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.089930 5036 generic.go:356] "Generic (PLEG): container finished" podID="78d2f934-75ce-4f00-b94d-5ac07009ecd2" containerID="b416a34b3bb40f610917156fa617d2029bc65bfe5e377bd24821cfc35376bf87" exitCode=0 Oct 30 22:34:10.090019 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.089991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" event={"ID":"78d2f934-75ce-4f00-b94d-5ac07009ecd2","Type":"ContainerDied","Data":"b416a34b3bb40f610917156fa617d2029bc65bfe5e377bd24821cfc35376bf87"} Oct 30 22:34:10.218125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.218089 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 22:34:10.234235 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.234201 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 22:34:10.836103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.836068 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 22:34:10.836393 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:10.836363 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" containerID="cri-o://cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76" gracePeriod=30 Oct 30 22:34:11.272158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.272133 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:11.443973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.443941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\") pod \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " Oct 30 22:34:11.443973 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.443980 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\") pod \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " Oct 30 22:34:11.444288 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.444128 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\" (UniqueName: \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\") pod \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\" (UID: \"78d2f934-75ce-4f00-b94d-5ac07009ecd2\") " Oct 30 22:34:11.444511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.444470 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca" pod "78d2f934-75ce-4f00-b94d-5ac07009ecd2" (UID: "78d2f934-75ce-4f00-b94d-5ac07009ecd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:11.444729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.444708 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.446546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.446521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm" pod "78d2f934-75ce-4f00-b94d-5ac07009ecd2" (UID: "78d2f934-75ce-4f00-b94d-5ac07009ecd2"). InnerVolumeSpecName "kube-api-access-h6gvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:11.446673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.446584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca" pod "78d2f934-75ce-4f00-b94d-5ac07009ecd2" (UID: "78d2f934-75ce-4f00-b94d-5ac07009ecd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:11.545595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.545558 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d2f934-75ce-4f00-b94d-5ac07009ecd2-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.545595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.545590 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6gvm\" (UniqueName: \"kubernetes.io/projected/78d2f934-75ce-4f00-b94d-5ac07009ecd2-kube-api-access-h6gvm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.648223 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.648184 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 22:34:11.648837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:11.648800 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" containerID="cri-o://1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5" gracePeriod=30 Oct 30 22:34:12.122592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:12.122560 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" Oct 30 22:34:12.122885 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:12.122588 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f" event={"ID":"78d2f934-75ce-4f00-b94d-5ac07009ecd2","Type":"ContainerDied","Data":"6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5"} Oct 30 22:34:12.123006 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:12.122920 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ced109204fbc7278f9a9aa9e3e767dc905cbc835bbc362ac675117b62e7bff5" Oct 30 22:34:13.134188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.134155 5036 generic.go:356] "Generic (PLEG): container finished" podID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerID="dff8110a3d121f5d4c0de089da739589cfe537546895ca4ab2accd7cdc3c4fc1" exitCode=0 Oct 30 22:34:13.134788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.134248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerDied","Data":"dff8110a3d121f5d4c0de089da739589cfe537546895ca4ab2accd7cdc3c4fc1"} Oct 30 22:34:13.134788 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.134340 5036 scope.go:122] "RemoveContainer" containerID="980ea0b135631f8ead6413179f1fbf50ab864b5317440acd590c999ba20dd928" Oct 30 22:34:13.239614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.239589 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:34:13.268836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.268809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.268844 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.268916 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.268954 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269000 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.268983 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269061 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269096 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269127 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\") pod \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\" (UID: \"4756f7ea-0fc6-45c9-8bb1-91e521a887ef\") " Oct 30 22:34:13.269563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269326 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:13.269563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:13.269563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269549 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:13.269740 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269566 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:13.269821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:13.269922 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269907 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.269986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269929 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.269986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269945 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.269986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269958 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.269986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.269971 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.272394 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.272282 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:13.272533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.272422 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "kube-api-access-pnz8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:13.272533 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.272453 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:13.272951 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.272923 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token" pod "4756f7ea-0fc6-45c9-8bb1-91e521a887ef" (UID: "4756f7ea-0fc6-45c9-8bb1-91e521a887ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:13.370848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.370752 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnz8m\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-kube-api-access-pnz8m\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.370848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.370785 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.370848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.370801 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.370848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.370814 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756f7ea-0fc6-45c9-8bb1-91e521a887ef-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.652050 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.652024 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:34:13.674386 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.674350 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\" (UniqueName: \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\") pod \"1c463d01-e96e-43fe-a278-beb0b28b2c88\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " Oct 30 22:34:13.674562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.674413 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\") pod \"1c463d01-e96e-43fe-a278-beb0b28b2c88\" (UID: \"1c463d01-e96e-43fe-a278-beb0b28b2c88\") " Oct 30 22:34:13.674789 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.674768 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume" pod "1c463d01-e96e-43fe-a278-beb0b28b2c88" (UID: "1c463d01-e96e-43fe-a278-beb0b28b2c88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:13.675031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.674993 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c463d01-e96e-43fe-a278-beb0b28b2c88-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.677042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.677004 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6" pod "1c463d01-e96e-43fe-a278-beb0b28b2c88" (UID: "1c463d01-e96e-43fe-a278-beb0b28b2c88"). InnerVolumeSpecName "kube-api-access-z2tz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:13.776097 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:13.776062 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tz6\" (UniqueName: \"kubernetes.io/projected/1c463d01-e96e-43fe-a278-beb0b28b2c88-kube-api-access-z2tz6\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:14.145163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.145126 5036 generic.go:356] "Generic (PLEG): container finished" podID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerID="6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821" exitCode=0 Oct 30 22:34:14.145678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.145181 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" Oct 30 22:34:14.145678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.145202 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" event={"ID":"1c463d01-e96e-43fe-a278-beb0b28b2c88","Type":"ContainerDied","Data":"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821"} Oct 30 22:34:14.145678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.145285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65" event={"ID":"1c463d01-e96e-43fe-a278-beb0b28b2c88","Type":"ContainerDied","Data":"366b909bddae61719b7b7adeb4508e4ab409734d2fc0e4774a470dbaecd94391"} Oct 30 22:34:14.145678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.145321 5036 scope.go:122] "RemoveContainer" containerID="6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821" Oct 30 22:34:14.147135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.147051 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" Oct 30 22:34:14.147135 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.147083 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn" event={"ID":"4756f7ea-0fc6-45c9-8bb1-91e521a887ef","Type":"ContainerDied","Data":"dff16bd22487fccc052457714404c51ffec9cd95bf155003f23e9a74a4db9bda"} Oct 30 22:34:14.157972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.157953 5036 scope.go:122] "RemoveContainer" containerID="6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821" Oct 30 22:34:14.158320 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:14.158247 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821\": container with ID starting with 6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821 not found: ID does not exist" containerID="6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821" Oct 30 22:34:14.158424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.158332 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821"} err="failed to get container status \"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821\": rpc error: code = NotFound desc = could not find container \"6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821\": container with ID starting with 6935514ec52ed0e3414b7736dcd6c44a192b8d5ce13a40033ee36980e8bf4821 not found: ID does not exist" Oct 30 22:34:14.158424 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.158355 5036 scope.go:122] "RemoveContainer" containerID="dff8110a3d121f5d4c0de089da739589cfe537546895ca4ab2accd7cdc3c4fc1" Oct 30 22:34:14.187821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.187793 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:34:14.189864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.189842 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-06ac2e-openshift-default-54699b8d6c-nvhhn"] Oct 30 22:34:14.201432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.201407 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:34:14.214450 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.214425 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0f73c9-6b587d7d7-8jw65"] Oct 30 22:34:14.499638 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.499545 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:14.499807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:14.499555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:15.218666 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.218624 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" containerID="cri-o://2126f21b0eb19c25a332a0a68bbf511f0380ab1bc035d30d24d543d11b0477cb" gracePeriod=25 Oct 30 22:34:15.234542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.234505 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" containerID="cri-o://825cbff6aa941e1d85716bab483284789a249718dcf69c964009e325b5f96d5d" gracePeriod=25 Oct 30 22:34:15.436017 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.435978 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:15.436159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.436051 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:15.509357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.509237 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:15.509357 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.509297 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:15.625589 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.625549 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 22:34:15.675363 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.675329 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 22:34:15.697599 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.697566 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\" (UniqueName: \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\") pod \"e8626147-8e13-47e6-81fa-29c398a5c843\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " Oct 30 22:34:15.697769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.697642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\") pod \"e8626147-8e13-47e6-81fa-29c398a5c843\" (UID: \"e8626147-8e13-47e6-81fa-29c398a5c843\") " Oct 30 22:34:15.697864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.697815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\" (UniqueName: \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\") pod \"1c29cd98-84db-4578-858e-e89b3de85bcd\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " Oct 30 22:34:15.697864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.697855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\") pod \"1c29cd98-84db-4578-858e-e89b3de85bcd\" (UID: \"1c29cd98-84db-4578-858e-e89b3de85bcd\") " Oct 30 22:34:15.698406 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.698380 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume" pod "e8626147-8e13-47e6-81fa-29c398a5c843" (UID: "e8626147-8e13-47e6-81fa-29c398a5c843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:15.698675 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.698652 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume" pod "1c29cd98-84db-4578-858e-e89b3de85bcd" (UID: "1c29cd98-84db-4578-858e-e89b3de85bcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:15.701480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.701454 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x" pod "e8626147-8e13-47e6-81fa-29c398a5c843" (UID: "e8626147-8e13-47e6-81fa-29c398a5c843"). InnerVolumeSpecName "kube-api-access-g562x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:15.701734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.701701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr" pod "1c29cd98-84db-4578-858e-e89b3de85bcd" (UID: "1c29cd98-84db-4578-858e-e89b3de85bcd"). InnerVolumeSpecName "kube-api-access-qcxpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:15.799719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.799616 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g562x\" (UniqueName: \"kubernetes.io/projected/e8626147-8e13-47e6-81fa-29c398a5c843-kube-api-access-g562x\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.799719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.799644 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8626147-8e13-47e6-81fa-29c398a5c843-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.799719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.799658 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcxpr\" (UniqueName: \"kubernetes.io/projected/1c29cd98-84db-4578-858e-e89b3de85bcd-kube-api-access-qcxpr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.799719 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.799673 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c29cd98-84db-4578-858e-e89b3de85bcd-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.894958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.894921 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj"] Oct 30 22:34:15.895473 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895458 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895474 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895487 5036 cpu_manager.go:414] "containerMap: removing container" podUID="78d2f934-75ce-4f00-b94d-5ac07009ecd2" containerName="revoke-keys" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895494 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d2f934-75ce-4f00-b94d-5ac07009ecd2" containerName="revoke-keys" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895508 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895513 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895538 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895542 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895551 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895556 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895565 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895570 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895648 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895655 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cd4692-c724-4687-b9cc-066b9c2c04d5" containerName="istio-proxy" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895665 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895671 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" containerName="payload-pre-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895677 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerName="payload-processing" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895684 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" containerName="istio-proxy" Oct 30 22:34:15.895742 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.895690 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d2f934-75ce-4f00-b94d-5ac07009ecd2" containerName="revoke-keys" Oct 30 22:34:15.899169 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.899143 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:15.905646 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.902976 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq"] Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.906808 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.907248 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.907359 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.908474 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.909329 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:15.919300 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.914944 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj"] Oct 30 22:34:15.943309 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:15.940515 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq"] Oct 30 22:34:16.002224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.002224 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002232 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9k4j\" (UniqueName: \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.002458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002316 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.002458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.002458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002364 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf5l\" (UniqueName: \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.002458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.002385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.097322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.097245 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 22:34:16.103983 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.103961 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.104098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.103997 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9k4j\" (UniqueName: \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.104098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104056 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.104098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104075 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.104292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104100 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf5l\" (UniqueName: \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.104292 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.104910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.105010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.104906 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.106510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.106484 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.106793 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.106773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.116809 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.116782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf5l\" (UniqueName: \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\") pod \"maas-api-revoke-sub-2bdbf661-zlvcj\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.117084 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.117063 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9k4j\" (UniqueName: \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\") pod \"maas-api-revoke-sub-a52f712f-28czq\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.170923 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.170890 5036 generic.go:356] "Generic (PLEG): container finished" podID="e8626147-8e13-47e6-81fa-29c398a5c843" containerID="1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9" exitCode=0 Oct 30 22:34:16.171142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.170952 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" Oct 30 22:34:16.171142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.170984 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" event={"ID":"e8626147-8e13-47e6-81fa-29c398a5c843","Type":"ContainerDied","Data":"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9"} Oct 30 22:34:16.171142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.171047 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9" event={"ID":"e8626147-8e13-47e6-81fa-29c398a5c843","Type":"ContainerDied","Data":"8bb7b4d586bcfa81ee1f277168b0c667e4bc159aaef310771d0853c237c24a19"} Oct 30 22:34:16.171142 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.171078 5036 scope.go:122] "RemoveContainer" containerID="1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9" Oct 30 22:34:16.172998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.172970 5036 generic.go:356] "Generic (PLEG): container finished" podID="1c29cd98-84db-4578-858e-e89b3de85bcd" containerID="e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02" exitCode=0 Oct 30 22:34:16.173168 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.173149 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" Oct 30 22:34:16.173525 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.173502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" event={"ID":"1c29cd98-84db-4578-858e-e89b3de85bcd","Type":"ContainerDied","Data":"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02"} Oct 30 22:34:16.173627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.173556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds" event={"ID":"1c29cd98-84db-4578-858e-e89b3de85bcd","Type":"ContainerDied","Data":"4912ed9be39293fc7c91b34bd8992a333984ec41a24693f7bafc8d24b6936f62"} Oct 30 22:34:16.175530 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.175509 5036 generic.go:356] "Generic (PLEG): container finished" podID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerID="cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76" exitCode=0 Oct 30 22:34:16.175647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.175601 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" event={"ID":"ce65d24c-4d03-4c1d-a614-760edd8f1c42","Type":"ContainerDied","Data":"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76"} Oct 30 22:34:16.175647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.175635 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" event={"ID":"ce65d24c-4d03-4c1d-a614-760edd8f1c42","Type":"ContainerDied","Data":"4758015d96738ba40d2cf0218a53ebc33bd32bb53993fac189915f411d7faab4"} Oct 30 22:34:16.175778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.175686 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx" Oct 30 22:34:16.186990 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.186926 5036 scope.go:122] "RemoveContainer" containerID="1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9" Oct 30 22:34:16.187381 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:16.187356 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9\": container with ID starting with 1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9 not found: ID does not exist" containerID="1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9" Oct 30 22:34:16.187499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.187388 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9"} err="failed to get container status \"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9\": rpc error: code = NotFound desc = could not find container \"1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9\": container with ID starting with 1f7f41557f8438dea1a0d1b24397455cf569bbe97d69a326e17e9a107b0487d9 not found: ID does not exist" Oct 30 22:34:16.187499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.187405 5036 scope.go:122] "RemoveContainer" containerID="e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02" Oct 30 22:34:16.191858 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.191836 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c463d01-e96e-43fe-a278-beb0b28b2c88" path="/var/lib/kubelet/pods/1c463d01-e96e-43fe-a278-beb0b28b2c88/volumes" Oct 30 22:34:16.192304 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.192254 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4756f7ea-0fc6-45c9-8bb1-91e521a887ef" path="/var/lib/kubelet/pods/4756f7ea-0fc6-45c9-8bb1-91e521a887ef/volumes" Oct 30 22:34:16.201989 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.201970 5036 scope.go:122] "RemoveContainer" containerID="e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02" Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:16.202500 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02\": container with ID starting with e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02 not found: ID does not exist" containerID="e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02" Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.202535 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02"} err="failed to get container status \"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02\": rpc error: code = NotFound desc = could not find container \"e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02\": container with ID starting with e82da891fbc8f1462f859e624673a5bdf668d950b400cf55f446576dd2b5bb02 not found: ID does not exist" Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.202557 5036 scope.go:122] "RemoveContainer" containerID="cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76" Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205336 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205409 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205445 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205515 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205638 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.206969 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.205862 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\") pod \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\" (UID: \"ce65d24c-4d03-4c1d-a614-760edd8f1c42\") " Oct 30 22:34:16.207845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.207061 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.207845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.207245 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.207845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.207529 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:16.207845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.207767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.208170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.207935 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.210868 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.210827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "kube-api-access-sxbf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.213026 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.212998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.214954 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.214932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.215070 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.215035 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 22:34:16.220651 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.220384 5036 scope.go:122] "RemoveContainer" containerID="cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76" Oct 30 22:34:16.221634 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.221609 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo" pod "ce65d24c-4d03-4c1d-a614-760edd8f1c42" (UID: "ce65d24c-4d03-4c1d-a614-760edd8f1c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:16.221772 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:16.221747 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76\": container with ID starting with cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76 not found: ID does not exist" containerID="cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76" Oct 30 22:34:16.221861 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.221782 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76"} err="failed to get container status \"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76\": rpc error: code = NotFound desc = could not find container \"cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76\": container with ID starting with cb8740d8ef6fb7667cc6ee8c7b44cd3e55915c95c113e5f83ccf5d8f9c23ff76 not found: ID does not exist" Oct 30 22:34:16.224369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.224344 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1529dc-6df768474c-rwqv9"] Oct 30 22:34:16.230458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.230434 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 22:34:16.236358 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.236332 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1529dc-68ffb4cc8c-lk2ds"] Oct 30 22:34:16.268294 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.258595 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:16.271893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.271851 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:16.307435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307403 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307435 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307437 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307453 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307470 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307488 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbf5\" (UniqueName: \"kubernetes.io/projected/ce65d24c-4d03-4c1d-a614-760edd8f1c42-kube-api-access-sxbf5\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307505 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307518 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307534 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce65d24c-4d03-4c1d-a614-760edd8f1c42-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.307633 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.307549 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce65d24c-4d03-4c1d-a614-760edd8f1c42-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.503915 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.503886 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 22:34:16.507775 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.507748 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-lf6sx"] Oct 30 22:34:16.579565 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.579508 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj"] Oct 30 22:34:16.584425 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.584390 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq"] Oct 30 22:34:16.949204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:16.949176 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 22:34:17.016229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016164 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016303 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016425 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016475 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.016603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016548 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.017023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016658 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.017023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\") pod \"5a609d12-a259-43b0-8247-362e5cec0ee8\" (UID: \"5a609d12-a259-43b0-8247-362e5cec0ee8\") " Oct 30 22:34:17.017023 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.016930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:17.017204 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017153 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:17.017632 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017605 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:17.017926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017860 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.017926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017887 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.017926 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017902 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.018226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.017997 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:17.018537 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.018499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:17.019496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.019466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:17.020225 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.020204 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "kube-api-access-qhr7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:17.020659 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.020637 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:17.022584 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.022555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy" pod "5a609d12-a259-43b0-8247-362e5cec0ee8" (UID: "5a609d12-a259-43b0-8247-362e5cec0ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118788 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118818 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118828 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhr7l\" (UniqueName: \"kubernetes.io/projected/5a609d12-a259-43b0-8247-362e5cec0ee8-kube-api-access-qhr7l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118839 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118848 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a609d12-a259-43b0-8247-362e5cec0ee8-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.118884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.118856 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a609d12-a259-43b0-8247-362e5cec0ee8-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:17.187773 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.187737 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" event={"ID":"9a22706a-ac70-4eef-8a10-eb40f764cd4d","Type":"ContainerStarted","Data":"063e142902b36e31fda33716646a2b2aeec4e5bfdf9c64c6c0edc2b463ae0ba7"} Oct 30 22:34:17.187997 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.187804 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" event={"ID":"9a22706a-ac70-4eef-8a10-eb40f764cd4d","Type":"ContainerStarted","Data":"442c2ad211123f270e1cdd9a67668e4b2818e438e89318c6730f6fbd47f09268"} Oct 30 22:34:17.189329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.189297 5036 generic.go:356] "Generic (PLEG): container finished" podID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerID="1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5" exitCode=0 Oct 30 22:34:17.189470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.189348 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" Oct 30 22:34:17.189470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.189370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" event={"ID":"5a609d12-a259-43b0-8247-362e5cec0ee8","Type":"ContainerDied","Data":"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5"} Oct 30 22:34:17.189470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.189417 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs" event={"ID":"5a609d12-a259-43b0-8247-362e5cec0ee8","Type":"ContainerDied","Data":"b30aa1b4bcffd55af2234014a63ad81ee5f77362dc1e62f55f56ca0d103be077"} Oct 30 22:34:17.189470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.189440 5036 scope.go:122] "RemoveContainer" containerID="1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5" Oct 30 22:34:17.192660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.192632 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" event={"ID":"2628436d-429b-4e67-bf3d-ac6a2f4a275c","Type":"ContainerStarted","Data":"0427ad798090e6b3bf6fbd6535923c4ad5ca56dcf76de6d4875fcf9af0b47323"} Oct 30 22:34:17.192777 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.192679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" event={"ID":"2628436d-429b-4e67-bf3d-ac6a2f4a275c","Type":"ContainerStarted","Data":"cf13c718c0d344c5e10b742089ee50c61265d649d7406c71a0310800e327a293"} Oct 30 22:34:17.206444 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.206426 5036 scope.go:122] "RemoveContainer" containerID="1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5" Oct 30 22:34:17.207722 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:17.207692 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5\": container with ID starting with 1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5 not found: ID does not exist" containerID="1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5" Oct 30 22:34:17.207825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.207731 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5"} err="failed to get container status \"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5\": rpc error: code = NotFound desc = could not find container \"1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5\": container with ID starting with 1a18dbb5428665782f45300b2c6bc73c3e83007ba50be8e7989b05af17e858c5 not found: ID does not exist" Oct 30 22:34:17.227105 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.227066 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" podStartSLOduration=2.227050454 podStartE2EDuration="2.227050454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:17.226123001 +0000 UTC m=+3281.176077958" watchObservedRunningTime="2026-10-30 22:34:17.227050454 +0000 UTC m=+3281.177005401" Oct 30 22:34:17.291725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.291666 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" podStartSLOduration=2.291651412 podStartE2EDuration="2.291651412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:17.276183712 +0000 UTC m=+3281.226138658" watchObservedRunningTime="2026-10-30 22:34:17.291651412 +0000 UTC m=+3281.241606375" Oct 30 22:34:17.320688 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.320658 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 22:34:17.351182 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.351148 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-cbkrs"] Oct 30 22:34:17.501233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:17.501190 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:18.103442 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:18.103404 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:34:18.187566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.187524 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c29cd98-84db-4578-858e-e89b3de85bcd" path="/var/lib/kubelet/pods/1c29cd98-84db-4578-858e-e89b3de85bcd/volumes" Oct 30 22:34:18.188077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.188054 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" path="/var/lib/kubelet/pods/5a609d12-a259-43b0-8247-362e5cec0ee8/volumes" Oct 30 22:34:18.188734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.188711 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" path="/var/lib/kubelet/pods/ce65d24c-4d03-4c1d-a614-760edd8f1c42/volumes" Oct 30 22:34:18.189477 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.189457 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8626147-8e13-47e6-81fa-29c398a5c843" path="/var/lib/kubelet/pods/e8626147-8e13-47e6-81fa-29c398a5c843/volumes" Oct 30 22:34:18.206229 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.206205 5036 generic.go:356] "Generic (PLEG): container finished" podID="2628436d-429b-4e67-bf3d-ac6a2f4a275c" containerID="0427ad798090e6b3bf6fbd6535923c4ad5ca56dcf76de6d4875fcf9af0b47323" exitCode=0 Oct 30 22:34:18.206370 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.206301 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" event={"ID":"2628436d-429b-4e67-bf3d-ac6a2f4a275c","Type":"ContainerDied","Data":"0427ad798090e6b3bf6fbd6535923c4ad5ca56dcf76de6d4875fcf9af0b47323"} Oct 30 22:34:18.208067 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.208047 5036 generic.go:356] "Generic (PLEG): container finished" podID="9a22706a-ac70-4eef-8a10-eb40f764cd4d" containerID="063e142902b36e31fda33716646a2b2aeec4e5bfdf9c64c6c0edc2b463ae0ba7" exitCode=0 Oct 30 22:34:18.208183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:18.208087 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" event={"ID":"9a22706a-ac70-4eef-8a10-eb40f764cd4d","Type":"ContainerDied","Data":"063e142902b36e31fda33716646a2b2aeec4e5bfdf9c64c6c0edc2b463ae0ba7"} Oct 30 22:34:19.395498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.395471 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:19.445705 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.445674 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:19.448650 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.448621 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\" (UniqueName: \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\") pod \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " Oct 30 22:34:19.448806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.448793 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\") pod \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " Oct 30 22:34:19.448848 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.448827 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\") pod \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\" (UID: \"2628436d-429b-4e67-bf3d-ac6a2f4a275c\") " Oct 30 22:34:19.451311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.451254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca" pod "2628436d-429b-4e67-bf3d-ac6a2f4a275c" (UID: "2628436d-429b-4e67-bf3d-ac6a2f4a275c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:19.451660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.451623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca" pod "2628436d-429b-4e67-bf3d-ac6a2f4a275c" (UID: "2628436d-429b-4e67-bf3d-ac6a2f4a275c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:19.453724 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.453699 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l" pod "2628436d-429b-4e67-bf3d-ac6a2f4a275c" (UID: "2628436d-429b-4e67-bf3d-ac6a2f4a275c"). InnerVolumeSpecName "kube-api-access-nqf5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:19.549621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.549532 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\" (UniqueName: \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\") pod \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " Oct 30 22:34:19.549621 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.549595 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\") pod \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " Oct 30 22:34:19.549851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.549626 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\") pod \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\" (UID: \"9a22706a-ac70-4eef-8a10-eb40f764cd4d\") " Oct 30 22:34:19.549980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.549948 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca" pod "9a22706a-ac70-4eef-8a10-eb40f764cd4d" (UID: "9a22706a-ac70-4eef-8a10-eb40f764cd4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:19.550278 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.550237 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqf5l\" (UniqueName: \"kubernetes.io/projected/2628436d-429b-4e67-bf3d-ac6a2f4a275c-kube-api-access-nqf5l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.550463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.550384 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.550507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.550479 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.550507 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.550497 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2628436d-429b-4e67-bf3d-ac6a2f4a275c-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.551963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.551934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca" pod "9a22706a-ac70-4eef-8a10-eb40f764cd4d" (UID: "9a22706a-ac70-4eef-8a10-eb40f764cd4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:19.551963 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.551951 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j" pod "9a22706a-ac70-4eef-8a10-eb40f764cd4d" (UID: "9a22706a-ac70-4eef-8a10-eb40f764cd4d"). InnerVolumeSpecName "kube-api-access-t9k4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:19.651942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.651900 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9k4j\" (UniqueName: \"kubernetes.io/projected/9a22706a-ac70-4eef-8a10-eb40f764cd4d-kube-api-access-t9k4j\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.651942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.651934 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a22706a-ac70-4eef-8a10-eb40f764cd4d-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:19.959054 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:19.959008 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" containerID="cri-o://6b78051d8be87faae2c07a5abbc4b1b95ee38cdef9f0bb2a022953123a9d5a4d" gracePeriod=45 Oct 30 22:34:20.229566 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.229541 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" Oct 30 22:34:20.229718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.229582 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52f712f-28czq" event={"ID":"9a22706a-ac70-4eef-8a10-eb40f764cd4d","Type":"ContainerDied","Data":"442c2ad211123f270e1cdd9a67668e4b2818e438e89318c6730f6fbd47f09268"} Oct 30 22:34:20.229718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.229633 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442c2ad211123f270e1cdd9a67668e4b2818e438e89318c6730f6fbd47f09268" Oct 30 22:34:20.231415 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.231392 5036 generic.go:356] "Generic (PLEG): container finished" podID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerID="6b78051d8be87faae2c07a5abbc4b1b95ee38cdef9f0bb2a022953123a9d5a4d" exitCode=0 Oct 30 22:34:20.231561 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.231520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" event={"ID":"787ab643-d777-4294-8bf2-7b3087c6ff02","Type":"ContainerDied","Data":"6b78051d8be87faae2c07a5abbc4b1b95ee38cdef9f0bb2a022953123a9d5a4d"} Oct 30 22:34:20.233276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.233242 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" Oct 30 22:34:20.233276 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.233249 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdbf661-zlvcj" event={"ID":"2628436d-429b-4e67-bf3d-ac6a2f4a275c","Type":"ContainerDied","Data":"cf13c718c0d344c5e10b742089ee50c61265d649d7406c71a0310800e327a293"} Oct 30 22:34:20.233432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.233308 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf13c718c0d344c5e10b742089ee50c61265d649d7406c71a0310800e327a293" Oct 30 22:34:20.235098 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.235010 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.267723 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\" (UniqueName: \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.267904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268153 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\" (UniqueName: \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268207 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268303 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\") pod \"787ab643-d777-4294-8bf2-7b3087c6ff02\" (UID: \"787ab643-d777-4294-8bf2-7b3087c6ff02\") " Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268394 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268568 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268839 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.268861 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.272289 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.269580 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.278365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.276666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.278365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.277064 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "kube-api-access-xp5qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:20.279039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.279003 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs" pod "787ab643-d777-4294-8bf2-7b3087c6ff02" (UID: "787ab643-d777-4294-8bf2-7b3087c6ff02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:20.370368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.370028 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.370368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.370053 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787ab643-d777-4294-8bf2-7b3087c6ff02-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.370368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.370064 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5qm\" (UniqueName: \"kubernetes.io/projected/787ab643-d777-4294-8bf2-7b3087c6ff02-kube-api-access-xp5qm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.370368 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.370074 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787ab643-d777-4294-8bf2-7b3087c6ff02-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.950257 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.950230 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:34:20.976576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.976545 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\") pod \"f02c42cb-3b4b-450a-941a-984a6b657906\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " Oct 30 22:34:20.976723 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.976613 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\" (UniqueName: \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\") pod \"f02c42cb-3b4b-450a-941a-984a6b657906\" (UID: \"f02c42cb-3b4b-450a-941a-984a6b657906\") " Oct 30 22:34:20.976986 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.976948 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume" pod "f02c42cb-3b4b-450a-941a-984a6b657906" (UID: "f02c42cb-3b4b-450a-941a-984a6b657906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:20.978972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:20.978947 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr" pod "f02c42cb-3b4b-450a-941a-984a6b657906" (UID: "f02c42cb-3b4b-450a-941a-984a6b657906"). InnerVolumeSpecName "kube-api-access-nmrjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:21.078609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.078571 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02c42cb-3b4b-450a-941a-984a6b657906-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:21.078609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.078603 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmrjr\" (UniqueName: \"kubernetes.io/projected/f02c42cb-3b4b-450a-941a-984a6b657906-kube-api-access-nmrjr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:21.245706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.245612 5036 generic.go:356] "Generic (PLEG): container finished" podID="f02c42cb-3b4b-450a-941a-984a6b657906" containerID="21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68" exitCode=0 Oct 30 22:34:21.245706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.245667 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" Oct 30 22:34:21.245947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.245712 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" event={"ID":"f02c42cb-3b4b-450a-941a-984a6b657906","Type":"ContainerDied","Data":"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68"} Oct 30 22:34:21.245947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.245783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc" event={"ID":"f02c42cb-3b4b-450a-941a-984a6b657906","Type":"ContainerDied","Data":"ba0bded85ce28bfd5e50977dc119638dd7af1557a79d1df44654efa6e5fa5c00"} Oct 30 22:34:21.245947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.245814 5036 scope.go:122] "RemoveContainer" containerID="21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68" Oct 30 22:34:21.247756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.247737 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" event={"ID":"787ab643-d777-4294-8bf2-7b3087c6ff02","Type":"ContainerDied","Data":"1f404080e266c55e30c4b335445d5bddec444c7a67038b553936a2bbe4143b91"} Oct 30 22:34:21.247838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.247799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws" Oct 30 22:34:21.261163 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.261148 5036 scope.go:122] "RemoveContainer" containerID="21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68" Oct 30 22:34:21.261490 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:21.261471 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68\": container with ID starting with 21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68 not found: ID does not exist" containerID="21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68" Oct 30 22:34:21.261539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.261501 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68"} err="failed to get container status \"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68\": rpc error: code = NotFound desc = could not find container \"21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68\": container with ID starting with 21f6dc54ffec8cdc4b48380d8921b4744e4350684c458a5c6f6444d9c1082b68 not found: ID does not exist" Oct 30 22:34:21.261539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.261518 5036 scope.go:122] "RemoveContainer" containerID="6b78051d8be87faae2c07a5abbc4b1b95ee38cdef9f0bb2a022953123a9d5a4d" Oct 30 22:34:21.276847 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.276810 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:34:21.280781 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.280758 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06ac2e-8688c5b7b7-2lbqc"] Oct 30 22:34:21.299497 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.299473 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 22:34:21.316219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:21.316192 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/simulator-w3-ceb36a-kserve-689b67bc4d-m2lws"] Oct 30 22:34:22.188837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:22.188803 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" path="/var/lib/kubelet/pods/787ab643-d777-4294-8bf2-7b3087c6ff02/volumes" Oct 30 22:34:22.189220 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:22.189205 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" path="/var/lib/kubelet/pods/f02c42cb-3b4b-450a-941a-984a6b657906/volumes" Oct 30 22:34:22.189542 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:22.189527 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f"] Oct 30 22:34:22.196483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:22.196457 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a39158-z5j4f"] Oct 30 22:34:24.189221 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:24.189179 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d2f934-75ce-4f00-b94d-5ac07009ecd2" path="/var/lib/kubelet/pods/78d2f934-75ce-4f00-b94d-5ac07009ecd2/volumes" Oct 30 22:34:25.509106 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:25.509071 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:25.509529 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:25.509072 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:26.733897 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:26.733867 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4"] Oct 30 22:34:26.751027 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:26.750996 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ceb36a-57df4"] Oct 30 22:34:28.190279 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:28.190233 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f4dd6e-747b-45d1-8b31-7caa99b56d8e" path="/var/lib/kubelet/pods/e5f4dd6e-747b-45d1-8b31-7caa99b56d8e/volumes" Oct 30 22:34:30.875369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:30.875337 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:34:30.875941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:30.875550 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" containerID="cri-o://b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f" gracePeriod=30 Oct 30 22:34:31.391702 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.391669 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b"] Oct 30 22:34:31.392181 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392169 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9a22706a-ac70-4eef-8a10-eb40f764cd4d" containerName="revoke-keys" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392183 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22706a-ac70-4eef-8a10-eb40f764cd4d" containerName="revoke-keys" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392195 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2628436d-429b-4e67-bf3d-ac6a2f4a275c" containerName="revoke-keys" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392200 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2628436d-429b-4e67-bf3d-ac6a2f4a275c" containerName="revoke-keys" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392222 5036 cpu_manager.go:414] "containerMap: removing container" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392227 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" Oct 30 22:34:31.392236 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392235 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392240 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392249 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392253 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392282 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392286 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392357 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="787ab643-d777-4294-8bf2-7b3087c6ff02" containerName="main" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392366 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a609d12-a259-43b0-8247-362e5cec0ee8" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392372 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce65d24c-4d03-4c1d-a614-760edd8f1c42" containerName="istio-proxy" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392380 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02c42cb-3b4b-450a-941a-984a6b657906" containerName="payload-pre-processing" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392387 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22706a-ac70-4eef-8a10-eb40f764cd4d" containerName="revoke-keys" Oct 30 22:34:31.392488 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.392393 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2628436d-429b-4e67-bf3d-ac6a2f4a275c" containerName="revoke-keys" Oct 30 22:34:31.397631 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.397609 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.414494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.414469 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b"] Oct 30 22:34:31.416717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.416694 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:34:31.416855 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.416740 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:31.431230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.431204 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:31.431399 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.431334 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:31.499329 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.499297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.499499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.499466 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.499558 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.499519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqd8l\" (UniqueName: \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.600449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.600412 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.600662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.600578 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.600662 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.600624 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqd8l\" (UniqueName: \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.601174 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.601136 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.603030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.603002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.615931 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.615908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqd8l\" (UniqueName: \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:31.712183 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:31.712097 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:32.000493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:32.000453 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b"] Oct 30 22:34:32.001339 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:34:32.001309 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03be61c7_b298_424a_b556_1ec7b418fc2f.slice/crio-5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8 WatchSource:0}: Error finding container 5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8: Status 404 returned error can't find the container with id 5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8 Oct 30 22:34:32.354934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:32.354895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" event={"ID":"03be61c7-b298-424a-b556-1ec7b418fc2f","Type":"ContainerStarted","Data":"5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8"} Oct 30 22:34:33.364962 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:33.364867 5036 generic.go:356] "Generic (PLEG): container finished" podID="03be61c7-b298-424a-b556-1ec7b418fc2f" containerID="b49f0ecf93ca58f27e47a1dc785949d3c62fd25e80c1f5841e4e9738b2005bc9" exitCode=0 Oct 30 22:34:33.365452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:33.364957 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" event={"ID":"03be61c7-b298-424a-b556-1ec7b418fc2f","Type":"ContainerDied","Data":"b49f0ecf93ca58f27e47a1dc785949d3c62fd25e80c1f5841e4e9738b2005bc9"} Oct 30 22:34:34.526572 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.526543 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:34.638233 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.638206 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\" (UniqueName: \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\") pod \"03be61c7-b298-424a-b556-1ec7b418fc2f\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " Oct 30 22:34:34.638431 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.638372 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\") pod \"03be61c7-b298-424a-b556-1ec7b418fc2f\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " Oct 30 22:34:34.638498 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.638485 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\") pod \"03be61c7-b298-424a-b556-1ec7b418fc2f\" (UID: \"03be61c7-b298-424a-b556-1ec7b418fc2f\") " Oct 30 22:34:34.638851 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.638822 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca" pod "03be61c7-b298-424a-b556-1ec7b418fc2f" (UID: "03be61c7-b298-424a-b556-1ec7b418fc2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:34.640544 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.640522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l" pod "03be61c7-b298-424a-b556-1ec7b418fc2f" (UID: "03be61c7-b298-424a-b556-1ec7b418fc2f"). InnerVolumeSpecName "kube-api-access-mqd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:34.640664 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.640584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca" pod "03be61c7-b298-424a-b556-1ec7b418fc2f" (UID: "03be61c7-b298-424a-b556-1ec7b418fc2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:34.739545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.739507 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:34.739545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.739540 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqd8l\" (UniqueName: \"kubernetes.io/projected/03be61c7-b298-424a-b556-1ec7b418fc2f-kube-api-access-mqd8l\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:34.739545 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:34.739552 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03be61c7-b298-424a-b556-1ec7b418fc2f-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:35.386909 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.386865 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" event={"ID":"03be61c7-b298-424a-b556-1ec7b418fc2f","Type":"ContainerDied","Data":"5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8"} Oct 30 22:34:35.387101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.386924 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed4445db0ca74c344e225950610279fd0db3f6fed8cb4a9ba6e1ba07db918f8" Oct 30 22:34:35.387101 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.386877 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b" Oct 30 22:34:35.389317 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.389246 5036 generic.go:356] "Generic (PLEG): container finished" podID="335e965e-3a7b-4dbc-9779-24f34148969a" containerID="825cbff6aa941e1d85716bab483284789a249718dcf69c964009e325b5f96d5d" exitCode=0 Oct 30 22:34:35.389467 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.389435 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" event={"ID":"335e965e-3a7b-4dbc-9779-24f34148969a","Type":"ContainerDied","Data":"825cbff6aa941e1d85716bab483284789a249718dcf69c964009e325b5f96d5d"} Oct 30 22:34:35.391678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.391652 5036 generic.go:356] "Generic (PLEG): container finished" podID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerID="2126f21b0eb19c25a332a0a68bbf511f0380ab1bc035d30d24d543d11b0477cb" exitCode=0 Oct 30 22:34:35.391836 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.391820 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" event={"ID":"1c3930c8-8d17-4300-bef9-5cf75dc767e6","Type":"ContainerDied","Data":"2126f21b0eb19c25a332a0a68bbf511f0380ab1bc035d30d24d543d11b0477cb"} Oct 30 22:34:35.505804 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.505773 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 22:34:35.602967 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.602940 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 22:34:35.650055 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.650008 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\" (UniqueName: \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\") pod \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " Oct 30 22:34:35.650218 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.650066 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\") pod \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\" (UID: \"1c3930c8-8d17-4300-bef9-5cf75dc767e6\") " Oct 30 22:34:35.650602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.650563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume" pod "1c3930c8-8d17-4300-bef9-5cf75dc767e6" (UID: "1c3930c8-8d17-4300-bef9-5cf75dc767e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:35.652492 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.652466 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh" pod "1c3930c8-8d17-4300-bef9-5cf75dc767e6" (UID: "1c3930c8-8d17-4300-bef9-5cf75dc767e6"). InnerVolumeSpecName "kube-api-access-gz8rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:35.751834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.751760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\" (UniqueName: \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\") pod \"335e965e-3a7b-4dbc-9779-24f34148969a\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " Oct 30 22:34:35.752025 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.751837 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\") pod \"335e965e-3a7b-4dbc-9779-24f34148969a\" (UID: \"335e965e-3a7b-4dbc-9779-24f34148969a\") " Oct 30 22:34:35.752115 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.752103 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz8rh\" (UniqueName: \"kubernetes.io/projected/1c3930c8-8d17-4300-bef9-5cf75dc767e6-kube-api-access-gz8rh\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:35.752185 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.752115 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3930c8-8d17-4300-bef9-5cf75dc767e6-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:35.752312 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.752255 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume" pod "335e965e-3a7b-4dbc-9779-24f34148969a" (UID: "335e965e-3a7b-4dbc-9779-24f34148969a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:35.754083 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.754064 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p" pod "335e965e-3a7b-4dbc-9779-24f34148969a" (UID: "335e965e-3a7b-4dbc-9779-24f34148969a"). InnerVolumeSpecName "kube-api-access-9s45p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:35.852998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.852963 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s45p\" (UniqueName: \"kubernetes.io/projected/335e965e-3a7b-4dbc-9779-24f34148969a-kube-api-access-9s45p\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:35.852998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:35.852990 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e965e-3a7b-4dbc-9779-24f34148969a-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.147039 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.147010 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:34:36.260715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260678 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.260905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260752 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.260905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260795 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.260905 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260845 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260916 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260956 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261060 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.260986 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:36.261514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261346 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:36.261514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261401 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:36.261514 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261495 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261835 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\") pod \"f4d298df-47a0-4a4c-a233-680e26b293df\" (UID: \"f4d298df-47a0-4a4c-a233-680e26b293df\") " Oct 30 22:34:36.261898 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261840 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:36.261991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.261957 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:36.262606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.262556 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.262606 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.262595 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d298df-47a0-4a4c-a233-680e26b293df-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.262872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.262611 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.262872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.262628 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.262872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.262644 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.263610 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.263579 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:36.263928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.263903 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:36.264715 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.264677 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:36.265197 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.265141 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c" pod "f4d298df-47a0-4a4c-a233-680e26b293df" (UID: "f4d298df-47a0-4a4c-a233-680e26b293df"). InnerVolumeSpecName "kube-api-access-dbn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:36.363866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.363780 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbn5c\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-kube-api-access-dbn5c\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.363866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.363811 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d298df-47a0-4a4c-a233-680e26b293df-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.363866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.363822 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d298df-47a0-4a4c-a233-680e26b293df-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.363866 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.363830 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d298df-47a0-4a4c-a233-680e26b293df-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.401838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.401809 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" Oct 30 22:34:36.401838 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.401829 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" event={"ID":"335e965e-3a7b-4dbc-9779-24f34148969a","Type":"ContainerDied","Data":"033d08e8857a07906679fc5568b6b1d1e8191f9a7b7fe35cdf3e3c92ce460ec5"} Oct 30 22:34:36.402108 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.401895 5036 scope.go:122] "RemoveContainer" containerID="825cbff6aa941e1d85716bab483284789a249718dcf69c964009e325b5f96d5d" Oct 30 22:34:36.403536 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.403511 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" Oct 30 22:34:36.403658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.403569 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4" event={"ID":"1c3930c8-8d17-4300-bef9-5cf75dc767e6","Type":"ContainerDied","Data":"e87e63b63c15b04f9f4c5e2853a0466c76e68b9867acf6e43596411b2a9cd46b"} Oct 30 22:34:36.405680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.405608 5036 generic.go:356] "Generic (PLEG): container finished" podID="f4d298df-47a0-4a4c-a233-680e26b293df" containerID="b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f" exitCode=0 Oct 30 22:34:36.405680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.405633 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" event={"ID":"f4d298df-47a0-4a4c-a233-680e26b293df","Type":"ContainerDied","Data":"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f"} Oct 30 22:34:36.405680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.405657 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" Oct 30 22:34:36.405865 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.405682 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk" event={"ID":"f4d298df-47a0-4a4c-a233-680e26b293df","Type":"ContainerDied","Data":"23fae5d3ba0391b0c2e61940c149eadc2ca3927d06e0eb75b8fad73be3f3ba30"} Oct 30 22:34:36.413906 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.413883 5036 scope.go:122] "RemoveContainer" containerID="2126f21b0eb19c25a332a0a68bbf511f0380ab1bc035d30d24d543d11b0477cb" Oct 30 22:34:36.424095 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.424075 5036 scope.go:122] "RemoveContainer" containerID="b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f" Oct 30 22:34:36.425668 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.425591 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 22:34:36.429852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.429831 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ceb36a-5bf6dd7df-xx4s4"] Oct 30 22:34:36.435227 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.435206 5036 scope.go:122] "RemoveContainer" containerID="b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f" Oct 30 22:34:36.435613 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:36.435585 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f\": container with ID starting with b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f not found: ID does not exist" containerID="b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f" Oct 30 22:34:36.435725 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.435622 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f"} err="failed to get container status \"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f\": rpc error: code = NotFound desc = could not find container \"b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f\": container with ID starting with b4c3ae8a701c0ec5466eb1119cbbdce6e4eda3b42a0a1eee2e7b63031dfcec0f not found: ID does not exist" Oct 30 22:34:36.439800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.439777 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 22:34:36.450790 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.450768 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg"] Oct 30 22:34:36.481872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.476477 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:34:36.488547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.488516 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-sjdvk"] Oct 30 22:34:36.509392 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:36.509360 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ceb36a-7777fdbc99-wqjhg" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.72:50052\" within 1s: context deadline exceeded" Oct 30 22:34:36.509498 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:34:36.509445 5036 logging.go:55] [core] [Channel #18501 SubChannel #18502]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.72:50052", ServerName: "10.132.0.72:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.72:50052: operation was canceled" Oct 30 22:34:37.360005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:37.359974 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:34:37.386449 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:37.386422 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:34:38.188112 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:38.188076 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" path="/var/lib/kubelet/pods/1c3930c8-8d17-4300-bef9-5cf75dc767e6/volumes" Oct 30 22:34:38.188499 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:38.188482 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" path="/var/lib/kubelet/pods/335e965e-3a7b-4dbc-9779-24f34148969a/volumes" Oct 30 22:34:38.188884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:38.188872 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" path="/var/lib/kubelet/pods/f4d298df-47a0-4a4c-a233-680e26b293df/volumes" Oct 30 22:34:41.563539 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:41.563504 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:34:42.375735 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:42.375701 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b"] Oct 30 22:34:42.382159 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:42.382128 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26a128-n2m9b"] Oct 30 22:34:44.187691 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:44.187654 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03be61c7-b298-424a-b556-1ec7b418fc2f" path="/var/lib/kubelet/pods/03be61c7-b298-424a-b556-1ec7b418fc2f/volumes" Oct 30 22:34:46.564065 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:46.564027 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" containerID="cri-o://66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801" gracePeriod=25 Oct 30 22:34:48.098488 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:34:48.098452 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:34:49.089749 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:49.089714 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:51.311942 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.311912 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75"] Oct 30 22:34:51.312766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312741 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03be61c7-b298-424a-b556-1ec7b418fc2f" containerName="revoke-keys" Oct 30 22:34:51.312766 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312770 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03be61c7-b298-424a-b556-1ec7b418fc2f" containerName="revoke-keys" Oct 30 22:34:51.312876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312807 5036 cpu_manager.go:414] "containerMap: removing container" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" Oct 30 22:34:51.312876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312817 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" Oct 30 22:34:51.312876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312858 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" Oct 30 22:34:51.312876 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312865 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" Oct 30 22:34:51.313012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312884 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" Oct 30 22:34:51.313012 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.312892 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" Oct 30 22:34:51.313078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.313015 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03be61c7-b298-424a-b556-1ec7b418fc2f" containerName="revoke-keys" Oct 30 22:34:51.313078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.313035 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="335e965e-3a7b-4dbc-9779-24f34148969a" containerName="payload-pre-processing" Oct 30 22:34:51.313078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.313049 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3930c8-8d17-4300-bef9-5cf75dc767e6" containerName="payload-processing" Oct 30 22:34:51.313078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.313061 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d298df-47a0-4a4c-a233-680e26b293df" containerName="istio-proxy" Oct 30 22:34:51.318811 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.318791 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.331595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.331475 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:51.331595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.331547 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:51.332005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.331975 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75"] Oct 30 22:34:51.369957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.369939 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:51.370228 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.370207 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:34:51.373654 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.373633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqvr\" (UniqueName: \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.373755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.373669 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.373840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.373823 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.474547 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.474509 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqvr\" (UniqueName: \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.474729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.474558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.474729 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.474704 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.475609 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.475525 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.477422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.477393 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.492869 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.492839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqvr\" (UniqueName: \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.638125 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.638099 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:51.925166 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:51.925136 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75"] Oct 30 22:34:51.927029 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:34:51.926993 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c451837_1d05_45db_aefe_85f79783200b.slice/crio-dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651 WatchSource:0}: Error finding container dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651: Status 404 returned error can't find the container with id dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651 Oct 30 22:34:52.547126 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:52.547090 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" event={"ID":"0c451837-1d05-45db-aefe-85f79783200b","Type":"ContainerStarted","Data":"0202cacdfed65ad5a8e0e7682447d442a1409f9c01f21803690aaf778b6c75c4"} Oct 30 22:34:52.547602 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:52.547140 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" event={"ID":"0c451837-1d05-45db-aefe-85f79783200b","Type":"ContainerStarted","Data":"dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651"} Oct 30 22:34:52.565832 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:52.565790 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" podStartSLOduration=1.565777253 podStartE2EDuration="1.565777253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.563454337 +0000 UTC m=+3316.513409284" watchObservedRunningTime="2026-10-30 22:34:52.565777253 +0000 UTC m=+3316.515732199" Oct 30 22:34:53.555704 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:53.555670 5036 generic.go:356] "Generic (PLEG): container finished" podID="0c451837-1d05-45db-aefe-85f79783200b" containerID="0202cacdfed65ad5a8e0e7682447d442a1409f9c01f21803690aaf778b6c75c4" exitCode=0 Oct 30 22:34:53.556160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:53.555747 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" event={"ID":"0c451837-1d05-45db-aefe-85f79783200b","Type":"ContainerDied","Data":"0202cacdfed65ad5a8e0e7682447d442a1409f9c01f21803690aaf778b6c75c4"} Oct 30 22:34:54.714703 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.714678 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:54.822562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.822474 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\") pod \"0c451837-1d05-45db-aefe-85f79783200b\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " Oct 30 22:34:54.822718 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.822627 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\") pod \"0c451837-1d05-45db-aefe-85f79783200b\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " Oct 30 22:34:54.822810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.822790 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\" (UniqueName: \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\") pod \"0c451837-1d05-45db-aefe-85f79783200b\" (UID: \"0c451837-1d05-45db-aefe-85f79783200b\") " Oct 30 22:34:54.823002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.822973 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca" pod "0c451837-1d05-45db-aefe-85f79783200b" (UID: "0c451837-1d05-45db-aefe-85f79783200b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:54.823179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.823161 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c451837-1d05-45db-aefe-85f79783200b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:54.824761 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.824733 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca" pod "0c451837-1d05-45db-aefe-85f79783200b" (UID: "0c451837-1d05-45db-aefe-85f79783200b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:54.825160 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.825132 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr" pod "0c451837-1d05-45db-aefe-85f79783200b" (UID: "0c451837-1d05-45db-aefe-85f79783200b"). InnerVolumeSpecName "kube-api-access-tqqvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:54.923927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.923891 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqvr\" (UniqueName: \"kubernetes.io/projected/0c451837-1d05-45db-aefe-85f79783200b-kube-api-access-tqqvr\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:54.923927 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:54.923922 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c451837-1d05-45db-aefe-85f79783200b-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:55.572234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:55.572203 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" Oct 30 22:34:55.572452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:55.572259 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75" event={"ID":"0c451837-1d05-45db-aefe-85f79783200b","Type":"ContainerDied","Data":"dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651"} Oct 30 22:34:55.572452 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:55.572326 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9747c5237dbd2ddb5ad31da71567451bafda5091e99dc0fc8cc03895c9f651" Oct 30 22:34:59.090005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:34:59.089971 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:01.446479 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:01.446444 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:35:02.082119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:02.082056 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75"] Oct 30 22:35:02.087031 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:02.087004 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-18ba2f-vmc75"] Oct 30 22:35:02.188340 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:02.188303 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c451837-1d05-45db-aefe-85f79783200b" path="/var/lib/kubelet/pods/0c451837-1d05-45db-aefe-85f79783200b/volumes" Oct 30 22:35:06.447458 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.447411 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" containerID="cri-o://1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c" gracePeriod=25 Oct 30 22:35:06.827774 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.827744 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:35:06.958219 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.958180 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\") pod \"2e8d351a-b512-45eb-b0be-a0755fb9816b\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " Oct 30 22:35:06.958461 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.958259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\" (UniqueName: \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\") pod \"2e8d351a-b512-45eb-b0be-a0755fb9816b\" (UID: \"2e8d351a-b512-45eb-b0be-a0755fb9816b\") " Oct 30 22:35:06.958595 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.958568 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume" pod "2e8d351a-b512-45eb-b0be-a0755fb9816b" (UID: "2e8d351a-b512-45eb-b0be-a0755fb9816b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:06.958845 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.958827 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e8d351a-b512-45eb-b0be-a0755fb9816b-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:06.960700 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:06.960674 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf" pod "2e8d351a-b512-45eb-b0be-a0755fb9816b" (UID: "2e8d351a-b512-45eb-b0be-a0755fb9816b"). InnerVolumeSpecName "kube-api-access-t8zlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:07.060551 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.060488 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8zlf\" (UniqueName: \"kubernetes.io/projected/2e8d351a-b512-45eb-b0be-a0755fb9816b-kube-api-access-t8zlf\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:07.665408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.665370 5036 generic.go:356] "Generic (PLEG): container finished" podID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerID="66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801" exitCode=0 Oct 30 22:35:07.665918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.665424 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" Oct 30 22:35:07.665918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.665470 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" event={"ID":"2e8d351a-b512-45eb-b0be-a0755fb9816b","Type":"ContainerDied","Data":"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801"} Oct 30 22:35:07.665918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.665542 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh" event={"ID":"2e8d351a-b512-45eb-b0be-a0755fb9816b","Type":"ContainerDied","Data":"b9549a287efcc9016ba2803f75f88fac0c4bec49a573c0e3b173b1086edadd29"} Oct 30 22:35:07.665918 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.665568 5036 scope.go:122] "RemoveContainer" containerID="66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801" Oct 30 22:35:07.677596 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.677573 5036 scope.go:122] "RemoveContainer" containerID="66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801" Oct 30 22:35:07.677887 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:07.677864 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801\": container with ID starting with 66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801 not found: ID does not exist" containerID="66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801" Oct 30 22:35:07.677977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.677893 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801"} err="failed to get container status \"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801\": rpc error: code = NotFound desc = could not find container \"66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801\": container with ID starting with 66ec74667c687ba3ee7ff39e57aa3e35cbb3b4316a64fca85b5a0a0569cf8801 not found: ID does not exist" Oct 30 22:35:07.688628 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.688601 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:35:07.692324 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:07.692302 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-26a128-7549d64655-27pgh"] Oct 30 22:35:08.188311 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:08.188253 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" path="/var/lib/kubelet/pods/2e8d351a-b512-45eb-b0be-a0755fb9816b/volumes" Oct 30 22:35:10.487380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487346 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw"] Oct 30 22:35:10.487839 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487824 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" Oct 30 22:35:10.487895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487841 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" Oct 30 22:35:10.487895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487862 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0c451837-1d05-45db-aefe-85f79783200b" containerName="revoke-keys" Oct 30 22:35:10.487895 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487868 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c451837-1d05-45db-aefe-85f79783200b" containerName="revoke-keys" Oct 30 22:35:10.488002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487952 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8d351a-b512-45eb-b0be-a0755fb9816b" containerName="payload-pre-processing" Oct 30 22:35:10.488002 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.487963 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c451837-1d05-45db-aefe-85f79783200b" containerName="revoke-keys" Oct 30 22:35:10.492785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.492763 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.514456 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.514430 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:10.514588 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.514470 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:10.514819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.514803 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:35:10.514947 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.514928 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:10.520494 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.520472 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw"] Oct 30 22:35:10.600604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.600571 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.600772 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.600630 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.600822 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.600790 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhxm\" (UniqueName: \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.701647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.701615 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.701824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.701678 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhxm\" (UniqueName: \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.701824 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.701781 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.702286 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.702248 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.704190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.704174 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.716226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.716204 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhxm\" (UniqueName: \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:10.806018 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:10.805939 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:11.088960 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:11.088929 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw"] Oct 30 22:35:11.090442 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:35:11.090413 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2ae636_8016_4ed3_92e4_dbc5301c448a.slice/crio-efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3 WatchSource:0}: Error finding container efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3: Status 404 returned error can't find the container with id efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3 Oct 30 22:35:11.706179 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:11.706101 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" event={"ID":"3a2ae636-8016-4ed3-92e4-dbc5301c448a","Type":"ContainerStarted","Data":"d258eb4454b783c27110674c2e81fce9c0d08e2c44965a8e0d314f5722792227"} Oct 30 22:35:11.706600 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:11.706181 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" event={"ID":"3a2ae636-8016-4ed3-92e4-dbc5301c448a","Type":"ContainerStarted","Data":"efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3"} Oct 30 22:35:11.724957 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:11.724900 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" podStartSLOduration=1.724883507 podStartE2EDuration="1.724883507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:11.723576795 +0000 UTC m=+3335.673531741" watchObservedRunningTime="2026-10-30 22:35:11.724883507 +0000 UTC m=+3335.674838454" Oct 30 22:35:12.715216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:12.715182 5036 generic.go:356] "Generic (PLEG): container finished" podID="3a2ae636-8016-4ed3-92e4-dbc5301c448a" containerID="d258eb4454b783c27110674c2e81fce9c0d08e2c44965a8e0d314f5722792227" exitCode=0 Oct 30 22:35:12.715673 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:12.715288 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" event={"ID":"3a2ae636-8016-4ed3-92e4-dbc5301c448a","Type":"ContainerDied","Data":"d258eb4454b783c27110674c2e81fce9c0d08e2c44965a8e0d314f5722792227"} Oct 30 22:35:13.882563 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:13.882534 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:14.046597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.046515 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\" (UniqueName: \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\") pod \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " Oct 30 22:35:14.046597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.046553 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\") pod \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " Oct 30 22:35:14.046597 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.046596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\") pod \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\" (UID: \"3a2ae636-8016-4ed3-92e4-dbc5301c448a\") " Oct 30 22:35:14.046972 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.046947 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca" pod "3a2ae636-8016-4ed3-92e4-dbc5301c448a" (UID: "3a2ae636-8016-4ed3-92e4-dbc5301c448a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:14.048852 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.048823 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca" pod "3a2ae636-8016-4ed3-92e4-dbc5301c448a" (UID: "3a2ae636-8016-4ed3-92e4-dbc5301c448a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:14.049020 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.049001 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm" pod "3a2ae636-8016-4ed3-92e4-dbc5301c448a" (UID: "3a2ae636-8016-4ed3-92e4-dbc5301c448a"). InnerVolumeSpecName "kube-api-access-lmhxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:14.148290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.148210 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhxm\" (UniqueName: \"kubernetes.io/projected/3a2ae636-8016-4ed3-92e4-dbc5301c448a-kube-api-access-lmhxm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:14.148290 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.148253 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:14.148623 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.148304 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2ae636-8016-4ed3-92e4-dbc5301c448a-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:14.731447 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.731418 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" Oct 30 22:35:14.731658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.731462 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw" event={"ID":"3a2ae636-8016-4ed3-92e4-dbc5301c448a","Type":"ContainerDied","Data":"efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3"} Oct 30 22:35:14.731658 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.731517 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efb54528377658ce7d8926f74ad93f8aed2b87a0211a4c918743b8e4579169f3" Oct 30 22:35:14.909397 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:14.909358 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:18.100021 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:18.099987 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:35:20.634010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:20.633971 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:35:21.806760 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:21.806731 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw"] Oct 30 22:35:21.812118 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:21.812088 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-38d6e2-wxhfw"] Oct 30 22:35:22.191433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:22.189624 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2ae636-8016-4ed3-92e4-dbc5301c448a" path="/var/lib/kubelet/pods/3a2ae636-8016-4ed3-92e4-dbc5301c448a/volumes" Oct 30 22:35:24.908769 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:24.908732 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:25.634764 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:25.634724 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" containerID="cri-o://0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf" gracePeriod=25 Oct 30 22:35:26.707495 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.707432 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:35:26.782077 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.781996 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\" (UniqueName: \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\") pod \"a050b519-eca1-4791-b519-84014afcb4bb\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " Oct 30 22:35:26.782244 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.782214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\") pod \"a050b519-eca1-4791-b519-84014afcb4bb\" (UID: \"a050b519-eca1-4791-b519-84014afcb4bb\") " Oct 30 22:35:26.782562 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.782528 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume" pod "a050b519-eca1-4791-b519-84014afcb4bb" (UID: "a050b519-eca1-4791-b519-84014afcb4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:26.784213 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.784191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm" pod "a050b519-eca1-4791-b519-84014afcb4bb" (UID: "a050b519-eca1-4791-b519-84014afcb4bb"). InnerVolumeSpecName "kube-api-access-64vbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:26.825524 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.825491 5036 generic.go:356] "Generic (PLEG): container finished" podID="a050b519-eca1-4791-b519-84014afcb4bb" containerID="1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c" exitCode=0 Oct 30 22:35:26.825750 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.825734 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" event={"ID":"a050b519-eca1-4791-b519-84014afcb4bb","Type":"ContainerDied","Data":"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c"} Oct 30 22:35:26.825801 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.825778 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" event={"ID":"a050b519-eca1-4791-b519-84014afcb4bb","Type":"ContainerDied","Data":"b6f3e764d808311479685768ab2f6f8cfe0bc6b9289c1ad603317de857ea2299"} Oct 30 22:35:26.825840 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.825801 5036 scope.go:122] "RemoveContainer" containerID="1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c" Oct 30 22:35:26.825945 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.825934 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j" Oct 30 22:35:26.838113 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.838091 5036 scope.go:122] "RemoveContainer" containerID="1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c" Oct 30 22:35:26.838574 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:26.838542 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c\": container with ID starting with 1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c not found: ID does not exist" containerID="1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c" Oct 30 22:35:26.838692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.838582 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c"} err="failed to get container status \"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c\": rpc error: code = NotFound desc = could not find container \"1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c\": container with ID starting with 1c9a77046a4d2ca8d6989e06838ccc80f1bb69cb7a80e5cb16c9503c6443b01c not found: ID does not exist" Oct 30 22:35:26.850605 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.850577 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:35:26.857475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.857450 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-18ba2f-85fc744bd9-2586j"] Oct 30 22:35:26.883881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.883845 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vbm\" (UniqueName: \"kubernetes.io/projected/a050b519-eca1-4791-b519-84014afcb4bb-kube-api-access-64vbm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:26.883881 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:26.883881 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a050b519-eca1-4791-b519-84014afcb4bb-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:28.187237 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:28.187201 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a050b519-eca1-4791-b519-84014afcb4bb" path="/var/lib/kubelet/pods/a050b519-eca1-4791-b519-84014afcb4bb/volumes" Oct 30 22:35:30.109603 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.109565 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9"] Oct 30 22:35:30.110044 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110030 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3a2ae636-8016-4ed3-92e4-dbc5301c448a" containerName="revoke-keys" Oct 30 22:35:30.110092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110045 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2ae636-8016-4ed3-92e4-dbc5301c448a" containerName="revoke-keys" Oct 30 22:35:30.110092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110054 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" Oct 30 22:35:30.110092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110060 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" Oct 30 22:35:30.110199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110142 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2ae636-8016-4ed3-92e4-dbc5301c448a" containerName="revoke-keys" Oct 30 22:35:30.110199 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.110151 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a050b519-eca1-4791-b519-84014afcb4bb" containerName="payload-processing" Oct 30 22:35:30.114216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.114198 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.130850 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.130824 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9"] Oct 30 22:35:30.130991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.130901 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:30.130991 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.130947 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:30.159321 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.159296 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:30.159454 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.159322 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:35:30.228627 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.228594 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.228893 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.228863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.229058 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.228917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpdq\" (UniqueName: \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.330834 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.330797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.331033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.330935 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.331033 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.330976 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpdq\" (UniqueName: \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.331770 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.331736 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.333517 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.333485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.344685 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.344657 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpdq\" (UniqueName: \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\") pod \"maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.427322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.427293 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:30.719913 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:35:30.719878 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfce3554_546b_485d_b905_6ccb53649ffd.slice/crio-218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176 WatchSource:0}: Error finding container 218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176: Status 404 returned error can't find the container with id 218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176 Oct 30 22:35:30.720920 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.720880 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9"] Oct 30 22:35:30.865103 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:30.865057 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" event={"ID":"bfce3554-546b-485d-b905-6ccb53649ffd","Type":"ContainerStarted","Data":"218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176"} Oct 30 22:35:31.874245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:31.874210 5036 generic.go:356] "Generic (PLEG): container finished" podID="bfce3554-546b-485d-b905-6ccb53649ffd" containerID="a454eb7a2c03d6548fa8b48b634038f9281fa792cc3c1700e0acf08bafcc4c00" exitCode=0 Oct 30 22:35:31.874706 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:31.874298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" event={"ID":"bfce3554-546b-485d-b905-6ccb53649ffd","Type":"ContainerDied","Data":"a454eb7a2c03d6548fa8b48b634038f9281fa792cc3c1700e0acf08bafcc4c00"} Oct 30 22:35:33.029463 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.029434 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:33.164022 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.163937 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\") pod \"bfce3554-546b-485d-b905-6ccb53649ffd\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " Oct 30 22:35:33.164206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.164049 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\" (UniqueName: \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\") pod \"bfce3554-546b-485d-b905-6ccb53649ffd\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " Oct 30 22:35:33.164206 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.164169 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\") pod \"bfce3554-546b-485d-b905-6ccb53649ffd\" (UID: \"bfce3554-546b-485d-b905-6ccb53649ffd\") " Oct 30 22:35:33.164579 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.164555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca" pod "bfce3554-546b-485d-b905-6ccb53649ffd" (UID: "bfce3554-546b-485d-b905-6ccb53649ffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:33.166403 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.166383 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq" pod "bfce3554-546b-485d-b905-6ccb53649ffd" (UID: "bfce3554-546b-485d-b905-6ccb53649ffd"). InnerVolumeSpecName "kube-api-access-5tpdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:33.166614 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.166589 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca" pod "bfce3554-546b-485d-b905-6ccb53649ffd" (UID: "bfce3554-546b-485d-b905-6ccb53649ffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:33.265913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.265871 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:33.265913 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.265908 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfce3554-546b-485d-b905-6ccb53649ffd-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:33.266144 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.265925 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tpdq\" (UniqueName: \"kubernetes.io/projected/bfce3554-546b-485d-b905-6ccb53649ffd-kube-api-access-5tpdq\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:33.890670 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.890641 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" Oct 30 22:35:33.890938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.890683 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9" event={"ID":"bfce3554-546b-485d-b905-6ccb53649ffd","Type":"ContainerDied","Data":"218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176"} Oct 30 22:35:33.890938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:33.890753 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="218fd51bc45e1878a661ea35611fcb713119881c92f4726a5ebbc53b5fde1176" Oct 30 22:35:34.608016 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:34.607972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:34.975342 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:34.975301 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 22:35:36.025821 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:36.025790 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw"] Oct 30 22:35:36.028778 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:36.028756 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df430947-567cw"] Oct 30 22:35:36.188746 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:36.188718 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702d1680-ebbd-4795-ad9e-5b88cfb9a1d9" path="/var/lib/kubelet/pods/702d1680-ebbd-4795-ad9e-5b88cfb9a1d9/volumes" Oct 30 22:35:37.027652 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:37.027622 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv"] Oct 30 22:35:37.035984 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:37.035956 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c3f31a-wrzpv"] Oct 30 22:35:38.187327 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:38.187296 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3add391c-b5c1-479e-8390-b79be8a98a5c" path="/var/lib/kubelet/pods/3add391c-b5c1-479e-8390-b79be8a98a5c/volumes" Oct 30 22:35:40.219418 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.219390 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 22:35:40.235470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.235438 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 22:35:40.264798 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.264766 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc"] Oct 30 22:35:40.265238 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.265225 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bfce3554-546b-485d-b905-6ccb53649ffd" containerName="revoke-keys" Oct 30 22:35:40.265313 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.265240 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfce3554-546b-485d-b905-6ccb53649ffd" containerName="revoke-keys" Oct 30 22:35:40.265408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.265396 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfce3554-546b-485d-b905-6ccb53649ffd" containerName="revoke-keys" Oct 30 22:35:40.268935 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.268915 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.273807 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.273781 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:40.276680 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.276641 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:40.277714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.276999 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:35:40.277714 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.277153 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:40.285408 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.285386 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc"] Oct 30 22:35:40.356888 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.356847 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.357052 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.356928 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.357131 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.357103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpsl\" (UniqueName: \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.458594 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.458555 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpsl\" (UniqueName: \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.458799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.458685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.458799 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.458733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.459365 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.459342 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.461226 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.461206 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.474730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.474678 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpsl\" (UniqueName: \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\") pod \"maas-api-revoke-sub-d85122a0-7vbzc\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.587369 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.587337 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:40.870563 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:35:40.870530 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cde7d6e_2844_4b71_bf02_4308133678a3.slice/crio-1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96 WatchSource:0}: Error finding container 1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96: Status 404 returned error can't find the container with id 1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96 Oct 30 22:35:40.871009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.870984 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc"] Oct 30 22:35:40.942323 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:40.942283 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" event={"ID":"9cde7d6e-2844-4b71-bf02-4308133678a3","Type":"ContainerStarted","Data":"1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96"} Oct 30 22:35:41.955234 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:41.955197 5036 generic.go:356] "Generic (PLEG): container finished" podID="9cde7d6e-2844-4b71-bf02-4308133678a3" containerID="89b714d888e49717a81bb5ba8f04725b7063d14974892d381db5afb023e68227" exitCode=0 Oct 30 22:35:41.955792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:41.955285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" event={"ID":"9cde7d6e-2844-4b71-bf02-4308133678a3","Type":"ContainerDied","Data":"89b714d888e49717a81bb5ba8f04725b7063d14974892d381db5afb023e68227"} Oct 30 22:35:43.114216 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.114192 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:43.288490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.288379 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\") pod \"9cde7d6e-2844-4b71-bf02-4308133678a3\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " Oct 30 22:35:43.288678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.288534 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\") pod \"9cde7d6e-2844-4b71-bf02-4308133678a3\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " Oct 30 22:35:43.288678 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.288642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\" (UniqueName: \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\") pod \"9cde7d6e-2844-4b71-bf02-4308133678a3\" (UID: \"9cde7d6e-2844-4b71-bf02-4308133678a3\") " Oct 30 22:35:43.289004 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.288971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca" pod "9cde7d6e-2844-4b71-bf02-4308133678a3" (UID: "9cde7d6e-2844-4b71-bf02-4308133678a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:43.289190 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.289167 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-openshift-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.290875 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.290826 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca" pod "9cde7d6e-2844-4b71-bf02-4308133678a3" (UID: "9cde7d6e-2844-4b71-bf02-4308133678a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:43.291053 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.291033 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl" pod "9cde7d6e-2844-4b71-bf02-4308133678a3" (UID: "9cde7d6e-2844-4b71-bf02-4308133678a3"). InnerVolumeSpecName "kube-api-access-jtpsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:43.390647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.390609 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cde7d6e-2844-4b71-bf02-4308133678a3-maas-api-xks-ca\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.390647 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.390641 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpsl\" (UniqueName: \"kubernetes.io/projected/9cde7d6e-2844-4b71-bf02-4308133678a3-kube-api-access-jtpsl\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.971611 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.971580 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" Oct 30 22:35:43.971827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.971618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85122a0-7vbzc" event={"ID":"9cde7d6e-2844-4b71-bf02-4308133678a3","Type":"ContainerDied","Data":"1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96"} Oct 30 22:35:43.971827 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:43.971666 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a30f5095c8612e832fbfa661779b72d5b3d38cb32858da16784ff3e4fb14f96" Oct 30 22:35:44.608149 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:44.608103 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:45.219676 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.219634 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" containerID="cri-o://8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548" gracePeriod=25 Oct 30 22:35:45.236078 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.236032 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" containerID="cri-o://910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894" gracePeriod=25 Oct 30 22:35:45.905433 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.905363 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:35:45.988061 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.988025 5036 generic.go:356] "Generic (PLEG): container finished" podID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerID="0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf" exitCode=0 Oct 30 22:35:45.988241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.988073 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" Oct 30 22:35:45.988241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.988115 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" event={"ID":"d65e758b-0bc0-46f3-87e6-751b7108599c","Type":"ContainerDied","Data":"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf"} Oct 30 22:35:45.988241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.988170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p" event={"ID":"d65e758b-0bc0-46f3-87e6-751b7108599c","Type":"ContainerDied","Data":"67df52100810a387a73561b63d7a5e4718cfb439e34e95c978eaba41d5a91e55"} Oct 30 22:35:45.988241 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:45.988194 5036 scope.go:122] "RemoveContainer" containerID="0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf" Oct 30 22:35:46.003550 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.003525 5036 scope.go:122] "RemoveContainer" containerID="0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf" Oct 30 22:35:46.004019 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:46.003986 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf\": container with ID starting with 0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf not found: ID does not exist" containerID="0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf" Oct 30 22:35:46.004132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.004030 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf"} err="failed to get container status \"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf\": rpc error: code = NotFound desc = could not find container \"0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf\": container with ID starting with 0b4ec3b31f50a745687c9b180a11a0242b29be6512d0738b62576d822f1834bf not found: ID does not exist" Oct 30 22:35:46.019977 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.019949 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\" (UniqueName: \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\") pod \"d65e758b-0bc0-46f3-87e6-751b7108599c\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " Oct 30 22:35:46.020081 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.020067 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\") pod \"d65e758b-0bc0-46f3-87e6-751b7108599c\" (UID: \"d65e758b-0bc0-46f3-87e6-751b7108599c\") " Oct 30 22:35:46.020455 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.020420 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume" pod "d65e758b-0bc0-46f3-87e6-751b7108599c" (UID: "d65e758b-0bc0-46f3-87e6-751b7108599c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:46.022121 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.022100 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q" pod "d65e758b-0bc0-46f3-87e6-751b7108599c" (UID: "d65e758b-0bc0-46f3-87e6-751b7108599c"). InnerVolumeSpecName "kube-api-access-5rg9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:46.121800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.121751 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65e758b-0bc0-46f3-87e6-751b7108599c-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:46.121800 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.121800 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rg9q\" (UniqueName: \"kubernetes.io/projected/d65e758b-0bc0-46f3-87e6-751b7108599c-kube-api-access-5rg9q\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:46.311049 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.310988 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:35:46.314432 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.314400 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-38d6e2-7cd9b6b94c-7bf7p"] Oct 30 22:35:46.521583 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:46.521495 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:47.036205 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:47.036175 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb"] Oct 30 22:35:47.040177 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:47.040145 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99582201-ffhwb"] Oct 30 22:35:47.532136 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:47.532101 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:48.098313 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:48.098275 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:35:48.187792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:48.187761 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af65d1cf-0e92-4ad7-8cc2-999c2e47b26d" path="/var/lib/kubelet/pods/af65d1cf-0e92-4ad7-8cc2-999c2e47b26d/volumes" Oct 30 22:35:48.188150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:48.188135 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" path="/var/lib/kubelet/pods/d65e758b-0bc0-46f3-87e6-751b7108599c/volumes" Oct 30 22:35:49.023941 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:49.023893 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b"] Oct 30 22:35:49.028378 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:49.028342 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2318e2af-mz72b"] Oct 30 22:35:49.997872 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:49.997833 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" containerID="cri-o://6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25" gracePeriod=45 Oct 30 22:35:50.188510 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.188475 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9cec06-7a60-4a6d-a2ea-eea4063436c6" path="/var/lib/kubelet/pods/df9cec06-7a60-4a6d-a2ea-eea4063436c6/volumes" Oct 30 22:35:50.256440 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.256373 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:35:50.264043 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264016 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264115 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264196 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\" (UniqueName: \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264359 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\") pod \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\" (UID: \"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a\") " Oct 30 22:35:50.264483 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.264787 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.264928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264831 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tmp-dir\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.264928 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.264836 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.266826 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.266793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.267207 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.267185 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "kube-api-access-d7562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:50.267326 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.267249 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs" pod "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" (UID: "08d8a0d7-4085-40f2-a750-d0c5d49b3b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:50.366230 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.366192 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-model-cache\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.366422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.366235 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7562\" (UniqueName: \"kubernetes.io/projected/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-kube-api-access-d7562\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.366422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.366252 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-dshm\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.366422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.366285 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-home\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.366422 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:50.366298 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a-tls-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.030337 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.030304 5036 generic.go:356] "Generic (PLEG): container finished" podID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerID="6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25" exitCode=0 Oct 30 22:35:51.030756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.030353 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" Oct 30 22:35:51.030756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.030389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" event={"ID":"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a","Type":"ContainerDied","Data":"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25"} Oct 30 22:35:51.030756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.030437 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s" event={"ID":"08d8a0d7-4085-40f2-a750-d0c5d49b3b0a","Type":"ContainerDied","Data":"1bf3347df3cb48f185d373b5f245315d439542104ac6883fbc1f933135ac7eec"} Oct 30 22:35:51.030756 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.030455 5036 scope.go:122] "RemoveContainer" containerID="6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25" Oct 30 22:35:51.045501 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.045481 5036 scope.go:122] "RemoveContainer" containerID="6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25" Oct 30 22:35:51.045762 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:35:51.045743 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25\": container with ID starting with 6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25 not found: ID does not exist" containerID="6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25" Oct 30 22:35:51.045825 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.045773 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25"} err="failed to get container status \"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25\": rpc error: code = NotFound desc = could not find container \"6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25\": container with ID starting with 6c6adf90329a75b67896da6efad82ca48fbee45b3248e6c84aa455a90023dd25 not found: ID does not exist" Oct 30 22:35:51.055139 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.055112 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 22:35:51.059361 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:51.059340 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/premium-w4-939a16-kserve-68655b5949-qch5s"] Oct 30 22:35:52.186490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:52.186458 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" path="/var/lib/kubelet/pods/08d8a0d7-4085-40f2-a750-d0c5d49b3b0a/volumes" Oct 30 22:35:55.552285 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:55.552219 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9"] Oct 30 22:35:55.560792 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:55.560760 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-939a16-q8lx9"] Oct 30 22:35:56.187968 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:56.187939 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfce3554-546b-485d-b905-6ccb53649ffd" path="/var/lib/kubelet/pods/bfce3554-546b-485d-b905-6ccb53649ffd/volumes" Oct 30 22:35:56.521904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:56.521810 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:57.531496 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:35:57.531457 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:02.319030 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:02.318992 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 22:36:02.319656 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:02.319297 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" containerID="cri-o://dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700" gracePeriod=30 Oct 30 22:36:03.663976 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:03.663943 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:03.664410 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:03.664015 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:03.757568 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:03.757539 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptctk_0127e352-a013-4fee-bd75-e074f768c1a6/manager/0.log" Oct 30 22:36:04.011352 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:04.011228 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dgpz6_7d32db30-79b4-4ff0-89d2-4f759c9e2cc4/manager/0.log" Oct 30 22:36:04.207371 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:04.207344 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-gldwq_ee31829b-82c1-40ee-8039-980b1c76e810/kuadrant-console-plugin/0.log" Oct 30 22:36:04.416837 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:04.416804 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mpqc_2add61fa-c2e9-4045-ae79-a20fb7eb3332/manager/0.log" Oct 30 22:36:05.490490 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.490422 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 22:36:05.525810 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.525787 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 22:36:05.636344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636308 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\") pod \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " Oct 30 22:36:05.636344 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636347 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\" (UniqueName: \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\") pod \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\" (UID: \"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3\") " Oct 30 22:36:05.636576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636389 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\") pod \"12167c7e-c89c-412b-ac41-509c30db9551\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " Oct 30 22:36:05.636576 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636415 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\" (UniqueName: \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\") pod \"12167c7e-c89c-412b-ac41-509c30db9551\" (UID: \"12167c7e-c89c-412b-ac41-509c30db9551\") " Oct 30 22:36:05.636762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636734 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume" pod "2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" (UID: "2c8a1e6f-330f-48ed-bac0-3265fe3e72c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:05.636819 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636756 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume" pod "12167c7e-c89c-412b-ac41-509c30db9551" (UID: "12167c7e-c89c-412b-ac41-509c30db9551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:05.636910 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636894 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:05.636974 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.636913 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12167c7e-c89c-412b-ac41-509c30db9551-plugins-config-volume\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:05.638734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.638707 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p" pod "12167c7e-c89c-412b-ac41-509c30db9551" (UID: "12167c7e-c89c-412b-ac41-509c30db9551"). InnerVolumeSpecName "kube-api-access-vm89p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:05.638862 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.638749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2" pod "2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" (UID: "2c8a1e6f-330f-48ed-bac0-3265fe3e72c3"). InnerVolumeSpecName "kube-api-access-wjtc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:05.738187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.738155 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtc2\" (UniqueName: \"kubernetes.io/projected/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3-kube-api-access-wjtc2\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:05.738187 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:05.738184 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm89p\" (UniqueName: \"kubernetes.io/projected/12167c7e-c89c-412b-ac41-509c30db9551-kube-api-access-vm89p\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:06.149005 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.148973 5036 generic.go:356] "Generic (PLEG): container finished" podID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerID="8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548" exitCode=0 Oct 30 22:36:06.149214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.149019 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" Oct 30 22:36:06.149214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.149060 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" event={"ID":"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3","Type":"ContainerDied","Data":"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548"} Oct 30 22:36:06.149214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.149113 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp" event={"ID":"2c8a1e6f-330f-48ed-bac0-3265fe3e72c3","Type":"ContainerDied","Data":"35a39c82fce0b1515029d8e976fb9dbfd6389107493279bbb16299f395e7b97a"} Oct 30 22:36:06.149214 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.149134 5036 scope.go:122] "RemoveContainer" containerID="8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548" Oct 30 22:36:06.150980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.150960 5036 generic.go:356] "Generic (PLEG): container finished" podID="12167c7e-c89c-412b-ac41-509c30db9551" containerID="910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894" exitCode=0 Oct 30 22:36:06.151082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.151004 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" event={"ID":"12167c7e-c89c-412b-ac41-509c30db9551","Type":"ContainerDied","Data":"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894"} Oct 30 22:36:06.151082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.151030 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" Oct 30 22:36:06.151082 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.151043 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh" event={"ID":"12167c7e-c89c-412b-ac41-509c30db9551","Type":"ContainerDied","Data":"00851ebfebfbc76cce64233106a819b803a65eca73a99a3c0b2ff6992f61a2ae"} Oct 30 22:36:06.162009 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.161992 5036 scope.go:122] "RemoveContainer" containerID="8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548" Oct 30 22:36:06.162319 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:36:06.162298 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548\": container with ID starting with 8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548 not found: ID does not exist" containerID="8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548" Oct 30 22:36:06.162382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.162326 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548"} err="failed to get container status \"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548\": rpc error: code = NotFound desc = could not find container \"8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548\": container with ID starting with 8dbe4a0cae075b2d34b8b25448d2c85a0d4ec99be2fc1a60d507c3c5b01a6548 not found: ID does not exist" Oct 30 22:36:06.162382 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.162342 5036 scope.go:122] "RemoveContainer" containerID="910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894" Oct 30 22:36:06.173820 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.173799 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 22:36:06.177667 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.174050 5036 scope.go:122] "RemoveContainer" containerID="910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894" Oct 30 22:36:06.178023 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:36:06.177996 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894\": container with ID starting with 910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894 not found: ID does not exist" containerID="910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894" Oct 30 22:36:06.178119 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.178034 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894"} err="failed to get container status \"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894\": rpc error: code = NotFound desc = could not find container \"910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894\": container with ID starting with 910dd413c0dd1e6b8692621680164ef7ba0fa7d5f2404ab95bcb639a6653d894 not found: ID does not exist" Oct 30 22:36:06.179380 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.179358 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-939a16-67bbd6c48f-795zp"] Oct 30 22:36:06.191730 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.191702 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" path="/var/lib/kubelet/pods/2c8a1e6f-330f-48ed-bac0-3265fe3e72c3/volumes" Oct 30 22:36:06.198046 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.198020 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 22:36:06.202211 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:06.202188 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-939a16-78b575c5f9-kkvnh"] Oct 30 22:36:07.570092 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.570067 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 22:36:07.660170 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660133 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660186 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660212 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660241 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660288 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660313 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660431 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660606 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660625 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.660744 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660654 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\") pod \"c62a658e-fda8-4360-b570-b2299f4c6b17\" (UID: \"c62a658e-fda8-4360-b570-b2299f4c6b17\") " Oct 30 22:36:07.661089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660751 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.661089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.660760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:07.661089 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.661071 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.661243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.661093 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-credential-socket\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.661243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.661108 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62a658e-fda8-4360-b570-b2299f4c6b17-istiod-ca-cert\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.661243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.661124 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-data\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.661243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.661139 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-workload-certs\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.662785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.662763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.662916 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.662894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.663258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.663234 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:07.663258 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.663238 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc" pod "c62a658e-fda8-4360-b570-b2299f4c6b17" (UID: "c62a658e-fda8-4360-b570-b2299f4c6b17"). InnerVolumeSpecName "kube-api-access-f8bjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.762578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.762547 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-podinfo\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.762578 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.762574 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bjc\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-kube-api-access-f8bjc\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.762780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.762586 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-token\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.762780 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:07.762595 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62a658e-fda8-4360-b570-b2299f4c6b17-istio-envoy\") on node \"ip-10-0-133-193.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:08.169938 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.169889 5036 generic.go:356] "Generic (PLEG): container finished" podID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerID="dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700" exitCode=0 Oct 30 22:36:08.170141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.169951 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" Oct 30 22:36:08.170141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.169960 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" event={"ID":"c62a658e-fda8-4360-b570-b2299f4c6b17","Type":"ContainerDied","Data":"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700"} Oct 30 22:36:08.170141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.170036 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6" event={"ID":"c62a658e-fda8-4360-b570-b2299f4c6b17","Type":"ContainerDied","Data":"514eb95e25aa4b10bc9b3a6ecff8c2ae215b9e606ca8c2eb50e0d03a168c4d7a"} Oct 30 22:36:08.170141 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.170067 5036 scope.go:122] "RemoveContainer" containerID="dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700" Oct 30 22:36:08.181707 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.181685 5036 scope.go:122] "RemoveContainer" containerID="dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700" Oct 30 22:36:08.182029 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:36:08.182007 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700\": container with ID starting with dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700 not found: ID does not exist" containerID="dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700" Oct 30 22:36:08.182132 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.182035 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700"} err="failed to get container status \"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700\": rpc error: code = NotFound desc = could not find container \"dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700\": container with ID starting with dcd15edcf59f163e381d03ac9feed1ee2692db272ece26ed09e6417c4e3d4700 not found: ID does not exist" Oct 30 22:36:08.186470 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.186448 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12167c7e-c89c-412b-ac41-509c30db9551" path="/var/lib/kubelet/pods/12167c7e-c89c-412b-ac41-509c30db9551/volumes" Oct 30 22:36:08.193090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.193067 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 22:36:08.200493 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:08.200471 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-9rvj6"] Oct 30 22:36:09.023692 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:09.023631 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z"] Oct 30 22:36:09.026475 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:09.026447 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f455a501-2mw9z"] Oct 30 22:36:10.186660 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:10.186625 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="637a2851-ef4e-40fb-8d75-c94c77fbce3d" path="/var/lib/kubelet/pods/637a2851-ef4e-40fb-8d75-c94c77fbce3d/volumes" Oct 30 22:36:10.187116 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:10.187099 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" path="/var/lib/kubelet/pods/c62a658e-fda8-4360-b570-b2299f4c6b17/volumes" Oct 30 22:36:18.098953 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:36:18.098913 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:36:20.024695 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:20.024662 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk"] Oct 30 22:36:20.030042 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:20.030016 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cadcd49-2qppk"] Oct 30 22:36:20.186953 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:20.186918 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ec71f9-e229-42e7-897e-9f6881c0c1e0" path="/var/lib/kubelet/pods/36ec71f9-e229-42e7-897e-9f6881c0c1e0/volumes" Oct 30 22:36:21.445322 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:21.445290 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-085509f7-zd2gx_548c7370-0a93-4c92-a9d6-859dcef4af0a/revoke-keys/0.log" Oct 30 22:36:21.624256 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:21.624225 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0c00211a-9wqsc_e5717723-7c1e-4bde-995e-0021e5ed5fe0/revoke-keys/0.log" Oct 30 22:36:21.807546 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:21.807470 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2bdbf661-zlvcj_2628436d-429b-4e67-bf3d-ac6a2f4a275c/revoke-keys/0.log" Oct 30 22:36:21.985934 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:21.985905 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3a16d9ed-t45n5_fff19f61-6c90-45c6-aecc-d07b3fd3adb3/revoke-keys/0.log" Oct 30 22:36:22.160785 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:22.160734 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-40106251-nmq72_8f7d47fb-c7da-4768-9e7e-544369f5fc5f/revoke-keys/0.log" Oct 30 22:36:22.348943 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:22.348892 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-52d023c1-j9wwc_d974d719-c312-426b-b99b-c6082de3a3d8/revoke-keys/0.log" Oct 30 22:36:22.522717 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:22.522629 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a52f712f-28czq_9a22706a-ac70-4eef-8a10-eb40f764cd4d/revoke-keys/0.log" Oct 30 22:36:22.698592 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:22.698561 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d85122a0-7vbzc_9cde7d6e-2844-4b71-bf02-4308133678a3/revoke-keys/0.log" Oct 30 22:36:23.077243 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:23.077212 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-edadd22e-fqqwv_939f7060-ddff-42d3-9423-cc23c7e8affd/revoke-keys/0.log" Oct 30 22:36:25.002806 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:25.002776 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptctk_0127e352-a013-4fee-bd75-e074f768c1a6/manager/0.log" Oct 30 22:36:25.201518 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:25.201483 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dgpz6_7d32db30-79b4-4ff0-89d2-4f759c9e2cc4/manager/0.log" Oct 30 22:36:25.381958 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:25.381930 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-gldwq_ee31829b-82c1-40ee-8039-980b1c76e810/kuadrant-console-plugin/0.log" Oct 30 22:36:25.586128 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:25.586101 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mpqc_2add61fa-c2e9-4045-ae79-a20fb7eb3332/manager/0.log" Oct 30 22:36:26.796420 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:26.796394 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896f5cvq7_e21da5c9-b269-4799-88ad-5bcd9251c418/istio-proxy/0.log" Oct 30 22:36:27.161210 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:27.161183 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ggxs5_724f765c-ab49-4da9-9ce2-f94dd12bb2b3/discovery/0.log" Oct 30 22:36:27.904090 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:27.904045 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-5bm7l_b9b6de7a-8d31-4f0e-990b-81525b0912d8/istio-proxy/1.log" Oct 30 22:36:28.500389 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:28.500358 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-c7wkl_d58ccfb1-e100-4d6f-9974-17dab6bbd846/istio-proxy/0.log" Oct 30 22:36:28.673209 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:28.673185 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-dl8xl_bf00519b-3c51-4867-bd8a-b68854743ae5/istio-proxy/0.log" Oct 30 22:36:29.265014 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:29.264987 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-744fc677d7-5x8pq_841d6863-c461-4be1-b810-eaeeb3977fba/istio-proxy/0.log" Oct 30 22:36:29.440581 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:29.440552 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-r65q6_5ccfc7e4-a88a-461d-9223-c1554dcaef70/payload-pre-processing/0.log" Oct 30 22:36:29.621272 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:29.621189 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9gpsn_fed55f70-e371-4b9a-9258-e70a781e919c/payload-processing/0.log" Oct 30 22:36:29.801520 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:29.801491 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df45874d9-82hkt_a98b79c6-cbd7-4d8a-ae09-4ecce2fdd98c/router/0.log" Oct 30 22:36:30.237245 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:30.237213 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-5bm7l_b9b6de7a-8d31-4f0e-990b-81525b0912d8/istio-proxy/0.log" Oct 30 22:36:30.944010 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:30.943981 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mm6v2_0b9abb0b-7989-467e-86a9-41bbe7ce3a59/main/0.log" Oct 30 22:36:31.319642 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:31.319558 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-w9r5n_0768611d-547a-42bd-b85b-65ae92070f3d/main/0.log" Oct 30 22:36:31.724472 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:31.724445 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9qcwk_e388b1ae-8405-44ab-949b-4d0d12da7c32/main/0.log" Oct 30 22:36:32.082150 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:32.082074 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-pdgbm_015c7202-4d5d-461c-99d1-cc3eca48e7b5/main/0.log" Oct 30 22:36:47.869956 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:47.869924 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nd669_92a37659-0e94-43e8-adb2-94ec2dfe09a1/global-pull-secret-syncer/0.log" Oct 30 22:36:48.086741 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:48.086710 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9jmb6_b589e26c-f56e-426a-9c25-396f65e0488f/connection-checker/0.log" Oct 30 22:36:48.099737 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:36:48.099706 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:36:48.122587 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:48.122531 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kwwwz_9b036611-6b37-40c6-9a5d-01e3f42d6e1e/connection-checker/0.log" Oct 30 22:36:48.155812 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:48.155791 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-r7rws_98c78665-fec7-466f-a032-51b497e7028a/connection-checker/0.log" Oct 30 22:36:48.426604 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:48.426573 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xfnx9_ce0f4cff-6fee-4e71-b5d1-381028a7f628/konnectivity-agent/0.log" Oct 30 22:36:48.497813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:48.497785 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-193.us-east-2.compute.internal_05953060daddf043bb6b311d0281468b/haproxy/0.log" Oct 30 22:36:51.414813 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:51.414778 5036 scope.go:122] "RemoveContainer" containerID="2e8468acccdeff011c579ae5afba49fc13950b799629195538afd7e660a3bbaa" Oct 30 22:36:51.425124 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:51.425104 5036 scope.go:122] "RemoveContainer" containerID="0bf2f4ff20e54128c123493ed3dbdce5f789fffe3db864db3aaa15f778c5b007" Oct 30 22:36:51.437417 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:51.437393 5036 scope.go:122] "RemoveContainer" containerID="f1d1869d44257ccf41ead1c7877ed234dc55db3b905cb8acfa61be7ec2e793b2" Oct 30 22:36:51.446644 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:51.446624 5036 scope.go:122] "RemoveContainer" containerID="87a544e386793e69fb7715a35f5e6004c21eb9d30ddcfc82835f2cbe7ef78b1b" Oct 30 22:36:56.059919 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:56.059884 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptctk_0127e352-a013-4fee-bd75-e074f768c1a6/manager/0.log" Oct 30 22:36:56.092515 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:56.092486 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dgpz6_7d32db30-79b4-4ff0-89d2-4f759c9e2cc4/manager/0.log" Oct 30 22:36:56.128933 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:56.128909 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-gldwq_ee31829b-82c1-40ee-8039-980b1c76e810/kuadrant-console-plugin/0.log" Oct 30 22:36:56.224188 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:56.224157 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mpqc_2add61fa-c2e9-4045-ae79-a20fb7eb3332/manager/0.log" Oct 30 22:36:58.487884 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:58.487853 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gfljx_dac1afb9-e804-406f-9064-8fe8e976e0c6/cluster-monitoring-operator/0.log" Oct 30 22:36:58.941982 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:58.941953 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvcg_1d4d1732-7920-4436-bb20-7c243aef4029/node-exporter/0.log" Oct 30 22:36:58.978998 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:58.978969 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvcg_1d4d1732-7920-4436-bb20-7c243aef4029/kube-rbac-proxy/0.log" Oct 30 22:36:59.010899 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:36:59.010873 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvcg_1d4d1732-7920-4436-bb20-7c243aef4029/init-textfile/0.log" Oct 30 22:37:00.222375 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222338 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh"] Oct 30 22:37:00.222864 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222846 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" Oct 30 22:37:00.222925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222867 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" Oct 30 22:37:00.222925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222885 5036 cpu_manager.go:414] "containerMap: removing container" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" Oct 30 22:37:00.222925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222890 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" Oct 30 22:37:00.222925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222901 5036 cpu_manager.go:414] "containerMap: removing container" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" Oct 30 22:37:00.222925 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222906 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222936 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222943 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222966 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222973 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222982 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9cde7d6e-2844-4b71-bf02-4308133678a3" containerName="revoke-keys" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.222988 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cde7d6e-2844-4b71-bf02-4308133678a3" containerName="revoke-keys" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223088 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65e758b-0bc0-46f3-87e6-751b7108599c" containerName="payload-pre-processing" Oct 30 22:37:00.223102 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223102 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62a658e-fda8-4360-b570-b2299f4c6b17" containerName="istio-proxy" Oct 30 22:37:00.223401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223110 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cde7d6e-2844-4b71-bf02-4308133678a3" containerName="revoke-keys" Oct 30 22:37:00.223401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223118 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a1e6f-330f-48ed-bac0-3265fe3e72c3" containerName="payload-processing" Oct 30 22:37:00.223401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223128 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="12167c7e-c89c-412b-ac41-509c30db9551" containerName="payload-pre-processing" Oct 30 22:37:00.223401 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.223136 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d8a0d7-4085-40f2-a750-d0c5d49b3b0a" containerName="main" Oct 30 22:37:00.228459 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.228438 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.234138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.234095 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k2j9d\"/\"default-dockercfg-xhph6\"" Oct 30 22:37:00.234138 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.234102 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"openshift-service-ca.crt\"" Oct 30 22:37:00.234351 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.234200 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"kube-root-ca.crt\"" Oct 30 22:37:00.234894 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.234870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh"] Oct 30 22:37:00.395762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.395724 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqfz\" (UniqueName: \"kubernetes.io/projected/505f0d14-499d-4c23-935b-6ea36aaaaa1e-kube-api-access-4pqfz\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.395762 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.395766 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-proc\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.395980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.395786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-lib-modules\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.395980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.395910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-podres\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.395980 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.395947 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-sys\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497182 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqfz\" (UniqueName: \"kubernetes.io/projected/505f0d14-499d-4c23-935b-6ea36aaaaa1e-kube-api-access-4pqfz\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-proc\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497291 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-lib-modules\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497329 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-podres\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-proc\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497371 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-sys\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497409 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-lib-modules\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497416 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-sys\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.497549 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.497435 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/505f0d14-499d-4c23-935b-6ea36aaaaa1e-podres\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.508709 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.508682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqfz\" (UniqueName: \"kubernetes.io/projected/505f0d14-499d-4c23-935b-6ea36aaaaa1e-kube-api-access-4pqfz\") pod \"perf-node-gather-daemonset-rtwgh\" (UID: \"505f0d14-499d-4c23-935b-6ea36aaaaa1e\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.543755 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.543732 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:00.827048 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:00.827012 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh"] Oct 30 22:37:00.829525 ip-10-0-133-193 kubenswrapper[5036]: W1030 22:37:00.829498 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod505f0d14_499d_4c23_935b_6ea36aaaaa1e.slice/crio-7cab8f70ab6cf0120b26174dc4b272c91fb79366fe6560d878e3bcd6b4bc8f21 WatchSource:0}: Error finding container 7cab8f70ab6cf0120b26174dc4b272c91fb79366fe6560d878e3bcd6b4bc8f21: Status 404 returned error can't find the container with id 7cab8f70ab6cf0120b26174dc4b272c91fb79366fe6560d878e3bcd6b4bc8f21 Oct 30 22:37:01.446176 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:01.446139 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-92mlp_e33c44f7-69f3-4f6e-bbfd-760ba6440f2c/networking-console-plugin/0.log" Oct 30 22:37:01.558645 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:01.558611 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" event={"ID":"505f0d14-499d-4c23-935b-6ea36aaaaa1e","Type":"ContainerStarted","Data":"98e5d009f27cd36ca473b178b42554fbd89667c3d64d681ab77e4f1513c25cf5"} Oct 30 22:37:01.558828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:01.558659 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" event={"ID":"505f0d14-499d-4c23-935b-6ea36aaaaa1e","Type":"ContainerStarted","Data":"7cab8f70ab6cf0120b26174dc4b272c91fb79366fe6560d878e3bcd6b4bc8f21"} Oct 30 22:37:01.558828 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:01.558712 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:01.577734 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:01.577689 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" podStartSLOduration=1.577672042 podStartE2EDuration="1.577672042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:01.575521869 +0000 UTC m=+3445.525476816" watchObservedRunningTime="2026-10-30 22:37:01.577672042 +0000 UTC m=+3445.527626991" Oct 30 22:37:02.437242 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:02.437203 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/2.log" Oct 30 22:37:02.441465 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:02.441440 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dljwb_9829f19b-65f8-4435-a942-0c997ee268c3/console-operator/3.log" Oct 30 22:37:04.132511 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:04.132479 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-zn459_675ec9b6-88b1-4fe3-a589-a0bcd26bfb38/volume-data-source-validator/0.log" Oct 30 22:37:06.105944 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:06.105909 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm8lm_4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/dns/0.log" Oct 30 22:37:06.137480 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:06.137457 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm8lm_4763a8d1-c6f0-49eb-81e3-a6919c5f4a85/kube-rbac-proxy/0.log" Oct 30 22:37:06.301064 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:06.301034 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h7xwb_5c533f37-a794-4b53-8779-86fbbb312bf6/dns-node-resolver/0.log" Oct 30 22:37:07.407904 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:07.407871 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgsq6_d6509794-be77-4085-aeec-92819c3c9041/node-ca/0.log" Oct 30 22:37:07.575407 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:07.575382 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-rtwgh" Oct 30 22:37:09.110310 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:09.110277 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896f5cvq7_e21da5c9-b269-4799-88ad-5bcd9251c418/istio-proxy/0.log" Oct 30 22:37:16.311476 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:16.311378 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896f5cvq7_e21da5c9-b269-4799-88ad-5bcd9251c418/istio-proxy/0.log" Oct 30 22:37:17.852158 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:17.852125 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896f5cvq7_e21da5c9-b269-4799-88ad-5bcd9251c418/istio-proxy/0.log" Oct 30 22:37:18.061215 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:18.061186 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896f5cvq7_e21da5c9-b269-4799-88ad-5bcd9251c418/istio-proxy/0.log" Oct 30 22:37:18.103120 ip-10-0-133-193 kubenswrapper[5036]: E1030 22:37:18.103040 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jqzzl" containerName="extractor" Oct 30 22:37:18.867372 ip-10-0-133-193 kubenswrapper[5036]: I1030 22:37:18.867338 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ggxs5_724f765c-ab49-4da9-9ce2-f94dd12bb2b3/discovery/0.log"